> The math and coding part is impressive but the agentic one is not.
I think this is very important to eventually become a viable replacement for coding models. Because most of the time coding harnesses are leveraging tool calls to gather the context and then write a solution.
I am hopeful, that one day we can replace Claude and OpenAI models with local SOTA LLMs
I see that going around, and either the test cases are too simplistic or I'm doing something wrong. I have a server with a 3090 in it, enough to run qwen3.6, but I haven't had much luck using it with either codex or oh-my-pi. They work, but the model gets really slow with ~64k context and the attention degrades quickly. You'll sometimes execute a prompt, the model will load a test file and say something like "I was presented with a test file but no command. What should I do with it?".
So yeah, while it's true that qwen3.6 is good for agentic coding, it's not very good for exploring the codebase and coming up with plans. You need to pair it today with a model capable of ingesting the whole context and providing a detailed plan, and even then the implementation might take 10x the amount of time it'd take for sonnet or Gemini 3 to crunch through the plan.
I can see that and I don't know your setup, but there are people pushing >70t/s with MTP on a single 3090, with big contexts still >50t/s. 64k is not a lot for agentic coding, and IIRC 128k with turboquant and the likes should be possible for you. r/LocalLLM/ and r/LocalLLaMA/ are worth a visit IMO.
You're not sharing what quantization you're using, in my experience, anything below Q8 and less than ~30B tends to basically be useless locally, at least for what you typically use codex et al for, I'm sure it works for very simple prompts.
But as soon as you go below Q8, the models get stuck in repeating loops, get the tool calling syntax wrong or just starts outputting gibberish after a short while.
That's absolutely possible, its just as we move towards more advancement, We'll soon see Small models being smart enough to not be judged by parameter count but their reasoning and intelligence. You can see examples like Qwen 3.6 27B.
Yeah this is key, a lot of people are still just looking at the number of params and thinking these models are toys. What Qwen 3.6 has shown is that reasoning and tool calling are just as important if not more.
I've been saying it for a long time now. I think small models are the future for LLMs. It's been fun seeing experiments to see just how much better models get by making them insanely large but it's not sustainable.
No I am not saying this model is a drop in Claude replacement. But I think in 2 years we might be really surprised what can be done in a desktop with commodity hardware, no connection to the internet, and a few models that span a subset of tasks.
Really happy to see amd put their hat in the ring. It's a good day for amd investors. I know a lot of AI bros will scoff at this, but having your first training run is a big deal for a new lab. AMD is on their way despite Nvidia having years of runway
using C was 100 times as productive as assembly. what happened was not that we finished software 100 times faster, but that we did projects 100 times bigger in the same time
same thing with smol local LLMs versus the big ones in the sky. your smol local LLM will only be able to tackle projects which are not comercially valuable anymore, because people expect 100x scope and features. which is fine as a hobby/art project
yes, we'll do amazing things with local LLMs in 2 years, but the big LLMs will do things beyond imagination (assembly vs C)
I disagree. I think people can make very good software by balancing their use of AI and their market knowledge. I still believe for the foreseeable future people can make wildly loved or mission critical software with 0 ai and have it be met with market interest.
I think we are going to see a surge in software claiming to do everything and becoming bloated and unsustainable.
I already see 1gpu local models 1 shotting games via vibe coding. I see people doing agentic programming, granted more slowly and cheaply than 12 Claude sessions.
The difference isn't as big as it was 2 months ago. In the past 45 days so many model releases have happened. Meanwhile frontier performance has stagnated and degraded. If it's a taste of what is to come I welcome it.
I'm like two months into a vibe coded C project. My issues are the same as ever. How to pack memory. What syscalls to run and when. Is the program stable after running for 24 hours? When I want to make a change it's usually a trade off with something else. There's no accounting for taste among humans. Let alone among an LM. It's great at implementing my ideas but terrible at coming up with those ideas. Architecture is always going to be king.
He could be right but time will tell if we can really achieve that level in open source space because as you know Even in open source space companies go closed when they achieve something really efficient and frontier. I'm not talking about all but that's usually a pattern
That is a danger, but for now it seems rather distant.
OpenAI has provided in the past a couple of open-weights models, but it does not seem to plan the release of any others.
But except for OpenAI and Anthropic, with this announcement Zyphra is the 12th company which has announced new improved open-weights models during the last couple of months.
A half of these 12 companies have launched not only small models with less than 128B parameters, but also big models with a number of parameters ranging from over 200B to over 1T.
So for now there is a healthy competition and the offerings in open-weights models are very diverse and numerous.
(The 12 directories on huggingface.co: deepseek-ai, google, ibm-granite, LiquidAI, MiniMaxAI, mistralai, moonshotai, nvidia, Qwen, XiaomiMiMo, zai-org, Zyphra.)
There are a lot of hats in the ring. I don't see Alibaba shutting down anytime soon. They make qwen.
Deepseek is doing valuations right now.
Moonshot is just getting started. Same with AMD. mistral is still working hard at it and has a customer base.
An Egyptian company dropped their first small model this month, Horus.
There are enough geopolitics at play that I expect this to be a very different outcome from typical startup market dynamics. If anything j worry about the big us labs longevity. The world is fed up with US tech it seems, and even for us citizens it's questionable the frontier labs have their interests in mind as they risk the entire economy.
> The math and coding part is impressive but the agentic one is not.
I think this is very important to eventually become a viable replacement for coding models. Because most of the time coding harnesses are leveraging tool calls to gather the context and then write a solution.
I am hopeful, that one day we can replace Claude and OpenAI models with local SOTA LLMs
It's pretty close already. Check qwen3.6 27b if you haven't already. People are vibe and agentic coding with it on a single GPU.
It is more finicky than Claude but if you hand hold it a bit it's crazy.
I see that going around, and either the test cases are too simplistic or I'm doing something wrong. I have a server with a 3090 in it, enough to run qwen3.6, but I haven't had much luck using it with either codex or oh-my-pi. They work, but the model gets really slow with ~64k context and the attention degrades quickly. You'll sometimes execute a prompt, the model will load a test file and say something like "I was presented with a test file but no command. What should I do with it?".
So yeah, while it's true that qwen3.6 is good for agentic coding, it's not very good for exploring the codebase and coming up with plans. You need to pair it today with a model capable of ingesting the whole context and providing a detailed plan, and even then the implementation might take 10x the amount of time it'd take for sonnet or Gemini 3 to crunch through the plan.
I can see that and I don't know your setup, but there are people pushing >70t/s with MTP on a single 3090, with big contexts still >50t/s. 64k is not a lot for agentic coding, and IIRC 128k with turboquant and the likes should be possible for you. r/LocalLLM/ and r/LocalLLaMA/ are worth a visit IMO.
EDIT: just found this recipe repo, may wanna give it a go: https://github.com/noonghunna/club-3090
You're not sharing what quantization you're using, in my experience, anything below Q8 and less than ~30B tends to basically be useless locally, at least for what you typically use codex et al for, I'm sure it works for very simple prompts.
But as soon as you go below Q8, the models get stuck in repeating loops, get the tool calling syntax wrong or just starts outputting gibberish after a short while.
I agree for planning it's not there yet. But I wouldn't be surprised if something came out that was in a similar weight class.
Try oh-my-openagent plan mode.
That's absolutely possible, its just as we move towards more advancement, We'll soon see Small models being smart enough to not be judged by parameter count but their reasoning and intelligence. You can see examples like Qwen 3.6 27B.
Yeah this is key, a lot of people are still just looking at the number of params and thinking these models are toys. What Qwen 3.6 has shown is that reasoning and tool calling are just as important if not more.
0.76 active and it's vaguely competitive at coding sounds promising.
LM studio doesn't let me actually run this yet though: "Unsupported safetensors format: null"
Announcement blogpost: https://www.zyphra.com/post/zaya1-8b
I've been saying it for a long time now. I think small models are the future for LLMs. It's been fun seeing experiments to see just how much better models get by making them insanely large but it's not sustainable.
No I am not saying this model is a drop in Claude replacement. But I think in 2 years we might be really surprised what can be done in a desktop with commodity hardware, no connection to the internet, and a few models that span a subset of tasks.
Really happy to see amd put their hat in the ring. It's a good day for amd investors. I know a lot of AI bros will scoff at this, but having your first training run is a big deal for a new lab. AMD is on their way despite Nvidia having years of runway
using C was 100 times as productive as assembly. what happened was not that we finished software 100 times faster, but that we did projects 100 times bigger in the same time
same thing with smol local LLMs versus the big ones in the sky. your smol local LLM will only be able to tackle projects which are not comercially valuable anymore, because people expect 100x scope and features. which is fine as a hobby/art project
yes, we'll do amazing things with local LLMs in 2 years, but the big LLMs will do things beyond imagination (assembly vs C)
I disagree. I think people can make very good software by balancing their use of AI and their market knowledge. I still believe for the foreseeable future people can make wildly loved or mission critical software with 0 ai and have it be met with market interest.
I think we are going to see a surge in software claiming to do everything and becoming bloated and unsustainable.
I already see 1gpu local models 1 shotting games via vibe coding. I see people doing agentic programming, granted more slowly and cheaply than 12 Claude sessions.
The difference isn't as big as it was 2 months ago. In the past 45 days so many model releases have happened. Meanwhile frontier performance has stagnated and degraded. If it's a taste of what is to come I welcome it.
I'm like two months into a vibe coded C project. My issues are the same as ever. How to pack memory. What syscalls to run and when. Is the program stable after running for 24 hours? When I want to make a change it's usually a trade off with something else. There's no accounting for taste among humans. Let alone among an LM. It's great at implementing my ideas but terrible at coming up with those ideas. Architecture is always going to be king.
You couldn't be any more right!
but he could be absolutely right
He could be right but time will tell if we can really achieve that level in open source space because as you know Even in open source space companies go closed when they achieve something really efficient and frontier. I'm not talking about all but that's usually a pattern
That is a danger, but for now it seems rather distant.
OpenAI has provided in the past a couple of open-weights models, but it does not seem to plan the release of any others.
But except for OpenAI and Anthropic, with this announcement Zyphra is the 12th company which has announced new improved open-weights models during the last couple of months.
A half of these 12 companies have launched not only small models with less than 128B parameters, but also big models with a number of parameters ranging from over 200B to over 1T.
So for now there is a healthy competition and the offerings in open-weights models are very diverse and numerous.
(The 12 directories on huggingface.co: deepseek-ai, google, ibm-granite, LiquidAI, MiniMaxAI, mistralai, moonshotai, nvidia, Qwen, XiaomiMiMo, zai-org, Zyphra.)
There are a lot of hats in the ring. I don't see Alibaba shutting down anytime soon. They make qwen.
Deepseek is doing valuations right now.
Moonshot is just getting started. Same with AMD. mistral is still working hard at it and has a customer base.
An Egyptian company dropped their first small model this month, Horus.
There are enough geopolitics at play that I expect this to be a very different outcome from typical startup market dynamics. If anything j worry about the big us labs longevity. The world is fed up with US tech it seems, and even for us citizens it's questionable the frontier labs have their interests in mind as they risk the entire economy.