That looks useful. I currently run OpenCode inside a containerd/Kata container and I see you support that backend. One thing I've been lazy with because it hasn't bit me yet is assuming the agent won't touch ".git", and diffing afterward. Being able to diff/apply modifications from the sandbox copy would be useful.
I did programming back in the days of MS-DOS. If you've got specific questions, I'm more than happy to follow up in replies.
I thought AI safety was something we had decades to worry about. I wasn't sure it could be solved, but never thought I'd really have to worry about it.
The thing that is frightening is how readily people accept the outputs, and just use them verbatim. Some, if not most, people would trust the output of the latest LLMs over that of actual experts in a field.
We really don't have to worry about a super-clever AI, as the hallucinations of the existing crop of LLMs are far more than enough to random-walk us into disaster.
It would be a far better situation if we had our OS house in order, as it is now, we don't have any widely used capability based secure OSs. I'm slightly hopeful that Genode will kick things off soon. It's been a long time coming.
The biggest danger we face right now is the cost of LLM inference exploding through the roof as the hidden subsidies end after the bubble pops. It's quite likely that a $20 or $200 per month subscription actually costs 5x that or more. If you've used LLMs to generate and maintain a code base that's exploded in size... you're going to have a much harder time after the price hikes hit.
AI safety is still an issue with bad AI. Not only do you have to protect against what it can do, but also protect against what it might accidentally do, the lies it tells that the masses believe, or people seeding the internet with misinformation specifically for AI to pick it up and regurgitate it to people without a proper source. It’s like money laundering, but for misinformation.
All these things are safety concerns of AI, but just rogue AGI.
I think for older programmers/CS academics, the second most surprising thing about the ChatGPT breakthrough in 2022 was that people cared about AI safety.
If you asking that do experienced programmers think AI safety experts (i.e. ai will doom the world) are overstating theie position the answer is that things are not the same and you cannot predict the future based on the past.
A much better approach would be to just see for yourself what you think. Use it and see how good/bad it is because anything you read or see on social media is likely just marketing.
Not sure what you're defining as an "older programer", but I've been coding for 20 years at this point and have been an AI doomer since around 2010, and hardcore AI doomer since 2015-2016ish...
I often wonder why I am so confident in my prediction of AI doom despite other intelligent people not being as convinced, and honestly I think it's just that I've been working with AI and building neural networks at a low level for well over decade.
I'm far from an expert when it comes to the frontier, but I feel like I have had to developed a strong intuition for these systems because it used to be fucking hard to get AI to do anything useful, but what you find is that the bottlenecks are literally always the same – compute, data and time
Algorithms matter, but mostly just because of efficiency.
For example, a convolutional neural network can very efficiently be trained to solve computer vision problems, but there's nothing stopping you using a much worse network architecture then throwing a ton more compute, data and time at the problem and getting the same result.
Algorithmic improvements are really just what we've needed to get around current compute, data and time bottlenecks. AIs like Deep Blue required a lot of algorithmic thinking back in the day to beat Kasparov in chess, but now we're so much less compute or data constrained that you could beat Kasparov without needing to put even a fraction of the algorithmic effort in if you just throw enough compute and data at it.
The reality is, AI doesn't fail. They may have appeared stupid to users for a long time but that wasn't because they didn't work, but because we were compute, data or time bottlenecked.
As soon as you assume compute and data will continue to grow, then it's really just a matter of time before all problems that are solvable with enough intelligence are solvable. Add significant algorithmic improvements every ~5 years to that equation and suddenly god-like AI seems like it's probably reasonably close.
The only question to ask then is what happens after you've created all these super-human AIs. And here I'm much less certain, but I see so many low probability risks in an ASI world that I'd bet everything on one of those bad things happening relatively soon after such AIs exist.
I suppose the hinge point is if algorithmic improvement + compute capacity is or exponential improvement. I flip/flop between the options as I listen to the Moonshot Podcast bros (lols because of obvious over-hype) and Lex Friedman's state of 2026 AI with Nathan Lambert and Sebastian Raschka.
If exponential, then yes I see more variance humanity's short/medium term outcome. More chance of doom, more chance of boom.
If logarithmic, well then improvements continue, but they will be incremental to what we have now and we will continue into some form of widespread agent adoption. Similar to wireless communication technology is prolific, but now taken 100% for granted.
Personally, even with Sol, the results I get from "stress test" one-shot prompts are abysmal. Hallucination of solutions to non-existent bugs as popped up multiple times.
So AGI take over, meh, low chance IMO.
I think one real risk is solo or sleeper cell type nefarious actors having a semi-competent engineering consultant which can enable small-scale weapons deployment.
Overall, I think the most probable short term risk is AI companions, hyper individualization and deterioration of social institutions. Much less flashy, but much more likely.
I mean look at the effects of social media on mental health. And that was only created 1 generation ago.
If it helps tune your probability weightings at all, almost every aspect of the field of AI is exponential and this has been true for decades.
20 years ago computers basically couldn't do voice or image recognition at all. 10 years ago it worked, but was still very flakey. Today it's super human.
I agree that if progress flattens out very rapidly and very soon then we'll at least have more time, but right now everything would suggest progress is still exponential and that flattening will now need to come quick and hard. This is largely why I've significantly increased my doom probability over the last year. I don't think there's any reasonable progress curve you can draw at this point that would suggest ASI isn't coming very soon. I suspect our only hope is nuclear war or a long tail good post-ASI outcome.
In regards to Sol, I think it's worth remembering that people were saying just a couple of years ago that AI can't even count the number of letters in a word or do basic maths. The fact the bar has risen so much we're saying they're still meh because they're not one-shotting problems is at least worth noting.
> I think one real risk is solo or sleeper cell type nefarious actors having a semi-competent engineering consultant which can enable small-scale weapons deployment.
Yep, the problem with AI isn't that there's any one risk that anyone can reasonably predict with high probability, but that there are many potential risks with signal digit probabilities. When you consider the overall risk landscape, that's when things get worrying.
> So AGI take over, meh, low chance IMO.
Right now I'd put this at like 10-20% probability. It seems unlikely to me, but quite possible if AIs of the future develop their own goals and are sufficiently super human.
The biggest risk is collapse I think. I don't think the world is stable in a post-ASI world. When the magic wish box will grant me whatever wish I like, I might wish something bad. In my mind this is the biggest risk by far.
> it's worth remembering that people were saying just a couple of years ago that AI can't even count the number of letters in a word or do basic maths.
That's very true. I think LLM's have passed the Turing test and I remeber that used to be a very high bar. I remember being flabbergasted when chatgpt went public.
I guess I'm just not convinced self-improvement is a thing...yet
I think it comes down to an information theoretic question of: Can a LLM create information that is a projection outside it's data subspace? Something like can it increase it's "matrix rank" and not just create a dependent row. What ever metaphor... it's fun
> but that there are many potential risks with signal digit probabilities. When you consider the overall risk landscape, that's when things get worrying.
That's a good point too. The number of possible bad and good things goes up and every gets access to doing more complex things previously contained to someone's skill silo.
What do you mean specifically by safety? It seems everyone, young and old, is running AI in YOLO-mode with full system access.
It's why I built the yoloAI tool, in fact!
That looks useful. I currently run OpenCode inside a containerd/Kata container and I see you support that backend. One thing I've been lazy with because it hasn't bit me yet is assuming the agent won't touch ".git", and diffing afterward. Being able to diff/apply modifications from the sandbox copy would be useful.
I did programming back in the days of MS-DOS. If you've got specific questions, I'm more than happy to follow up in replies.
I thought AI safety was something we had decades to worry about. I wasn't sure it could be solved, but never thought I'd really have to worry about it.
The thing that is frightening is how readily people accept the outputs, and just use them verbatim. Some, if not most, people would trust the output of the latest LLMs over that of actual experts in a field.
We really don't have to worry about a super-clever AI, as the hallucinations of the existing crop of LLMs are far more than enough to random-walk us into disaster.
It would be a far better situation if we had our OS house in order, as it is now, we don't have any widely used capability based secure OSs. I'm slightly hopeful that Genode will kick things off soon. It's been a long time coming.
The biggest danger we face right now is the cost of LLM inference exploding through the roof as the hidden subsidies end after the bubble pops. It's quite likely that a $20 or $200 per month subscription actually costs 5x that or more. If you've used LLMs to generate and maintain a code base that's exploded in size... you're going to have a much harder time after the price hikes hit.
AI safety is still an issue with bad AI. Not only do you have to protect against what it can do, but also protect against what it might accidentally do, the lies it tells that the masses believe, or people seeding the internet with misinformation specifically for AI to pick it up and regurgitate it to people without a proper source. It’s like money laundering, but for misinformation.
All these things are safety concerns of AI, but just rogue AGI.
What are talking about?
I think for older programmers/CS academics, the second most surprising thing about the ChatGPT breakthrough in 2022 was that people cared about AI safety.
If you asking that do experienced programmers think AI safety experts (i.e. ai will doom the world) are overstating theie position the answer is that things are not the same and you cannot predict the future based on the past.
A much better approach would be to just see for yourself what you think. Use it and see how good/bad it is because anything you read or see on social media is likely just marketing.
I would use the word naïveté, but arrogance too, sure.
Define “older”, please.
Someone is older in late 30s. :)
Or did you mean “those who experienced mainframes”?
Thanks.
Not sure what you're defining as an "older programer", but I've been coding for 20 years at this point and have been an AI doomer since around 2010, and hardcore AI doomer since 2015-2016ish...
I often wonder why I am so confident in my prediction of AI doom despite other intelligent people not being as convinced, and honestly I think it's just that I've been working with AI and building neural networks at a low level for well over decade.
I'm far from an expert when it comes to the frontier, but I feel like I have had to developed a strong intuition for these systems because it used to be fucking hard to get AI to do anything useful, but what you find is that the bottlenecks are literally always the same – compute, data and time
Algorithms matter, but mostly just because of efficiency.
For example, a convolutional neural network can very efficiently be trained to solve computer vision problems, but there's nothing stopping you using a much worse network architecture then throwing a ton more compute, data and time at the problem and getting the same result.
Algorithmic improvements are really just what we've needed to get around current compute, data and time bottlenecks. AIs like Deep Blue required a lot of algorithmic thinking back in the day to beat Kasparov in chess, but now we're so much less compute or data constrained that you could beat Kasparov without needing to put even a fraction of the algorithmic effort in if you just throw enough compute and data at it.
The reality is, AI doesn't fail. They may have appeared stupid to users for a long time but that wasn't because they didn't work, but because we were compute, data or time bottlenecked.
As soon as you assume compute and data will continue to grow, then it's really just a matter of time before all problems that are solvable with enough intelligence are solvable. Add significant algorithmic improvements every ~5 years to that equation and suddenly god-like AI seems like it's probably reasonably close.
The only question to ask then is what happens after you've created all these super-human AIs. And here I'm much less certain, but I see so many low probability risks in an ASI world that I'd bet everything on one of those bad things happening relatively soon after such AIs exist.
I suppose the hinge point is if algorithmic improvement + compute capacity is or exponential improvement. I flip/flop between the options as I listen to the Moonshot Podcast bros (lols because of obvious over-hype) and Lex Friedman's state of 2026 AI with Nathan Lambert and Sebastian Raschka.
If exponential, then yes I see more variance humanity's short/medium term outcome. More chance of doom, more chance of boom.
If logarithmic, well then improvements continue, but they will be incremental to what we have now and we will continue into some form of widespread agent adoption. Similar to wireless communication technology is prolific, but now taken 100% for granted.
Personally, even with Sol, the results I get from "stress test" one-shot prompts are abysmal. Hallucination of solutions to non-existent bugs as popped up multiple times.
So AGI take over, meh, low chance IMO.
I think one real risk is solo or sleeper cell type nefarious actors having a semi-competent engineering consultant which can enable small-scale weapons deployment.
Overall, I think the most probable short term risk is AI companions, hyper individualization and deterioration of social institutions. Much less flashy, but much more likely.
I mean look at the effects of social media on mental health. And that was only created 1 generation ago.
If it helps tune your probability weightings at all, almost every aspect of the field of AI is exponential and this has been true for decades.
20 years ago computers basically couldn't do voice or image recognition at all. 10 years ago it worked, but was still very flakey. Today it's super human.
I agree that if progress flattens out very rapidly and very soon then we'll at least have more time, but right now everything would suggest progress is still exponential and that flattening will now need to come quick and hard. This is largely why I've significantly increased my doom probability over the last year. I don't think there's any reasonable progress curve you can draw at this point that would suggest ASI isn't coming very soon. I suspect our only hope is nuclear war or a long tail good post-ASI outcome.
In regards to Sol, I think it's worth remembering that people were saying just a couple of years ago that AI can't even count the number of letters in a word or do basic maths. The fact the bar has risen so much we're saying they're still meh because they're not one-shotting problems is at least worth noting.
> I think one real risk is solo or sleeper cell type nefarious actors having a semi-competent engineering consultant which can enable small-scale weapons deployment.
Yep, the problem with AI isn't that there's any one risk that anyone can reasonably predict with high probability, but that there are many potential risks with signal digit probabilities. When you consider the overall risk landscape, that's when things get worrying.
> So AGI take over, meh, low chance IMO.
Right now I'd put this at like 10-20% probability. It seems unlikely to me, but quite possible if AIs of the future develop their own goals and are sufficiently super human.
The biggest risk is collapse I think. I don't think the world is stable in a post-ASI world. When the magic wish box will grant me whatever wish I like, I might wish something bad. In my mind this is the biggest risk by far.
> it's worth remembering that people were saying just a couple of years ago that AI can't even count the number of letters in a word or do basic maths.
That's very true. I think LLM's have passed the Turing test and I remeber that used to be a very high bar. I remember being flabbergasted when chatgpt went public.
I guess I'm just not convinced self-improvement is a thing...yet
I think it comes down to an information theoretic question of: Can a LLM create information that is a projection outside it's data subspace? Something like can it increase it's "matrix rank" and not just create a dependent row. What ever metaphor... it's fun
> but that there are many potential risks with signal digit probabilities. When you consider the overall risk landscape, that's when things get worrying.
That's a good point too. The number of possible bad and good things goes up and every gets access to doing more complex things previously contained to someone's skill silo.