We ran v4.1 Flash through our evaluations and found it to be smarter and faster than V4 Flash, with a commensurate price bump. Some notes:
- Because it's a heavy reasoner, it sits near Gemini 3.7 Flash on the Pareto front (not as cheap as the price suggests in practice).
- Closer than expected to the top open weights models (GLM 5.3 and Kimi K3) in agentic coding, at lower cost.
- Chinese models have always been strong iterators in an agentic harness. This model is no different, reaching an average percentile ~20% higher when given a harness vs a one-shot solution. That one-shot fluid intelligence is what makes a model feel smart, though, and typically results in fewer attempts/tokens to solve a problem, and American frontier models are still far ahead in that department.
The new architecture is interesting. It puts pricing between their old Flash and Pro lineups, suggesting they might be abandoning their super-cheap flash models (which weren't that fast due to heavy reasoning) and their pro models (which sort of flopped and weren't consistently better than their flash models, despite the size/cost) and shipping a strong intermediate that competes with the Gemini Flash series.
DeepSeek models have such good benchmark performance, amazing pricing, and the team over there seems to be widely considered impressive.
I just haven't found them to be very good? I've had a ton more success with the GLM models (since 5.2 anyway). Maybe I'm just holding it wrong, DS models seem to get stuck in loops or tell me nonsense. GLM feels like budget Claude.
My experience is complete opposite from yours. The V4 Flash was already quite good, but V4.1 is really very close to SOTA in my books. I've eval'd these models for weeks against Gemini 3.8, Kimi K3 and Opus 5, and DeepSeek absolutely wins these evals.
It's really crazy. We price per token our customers. If Kimi was about 30% of the price of Opus 5 for the same quality, DeepSeek is 1/10th of a price of Kimi K3. We've come down in price so much that I seriously cannot recommend other models before they reduce their pricing.
And what is really interesting is its programming ability. As I've said in my previous comments, I use agents a lot in my work. Since early Opus days until now I have 7-8 agents working in parallel for different tasks. Rust, design, GEPA, evals, analysis. For a long time Kimi K3 was the best model for this work, and before that GPT 5.5. But I still can't really believe how well DeepSeek works here. I really try to find faults from it, trying to see that it must be doing sloppy work and be worse than the others. But it does not. It finishes every task I give to it. And the cost per task is under a dollar, usually 15-30 cents.
In comparison the same task with Kimi would be 3-15 dollars; sometimes closing to 100. And before that with GPT 5.5 a 800 dollar task was not uncommon if I spent days evaluating models.
Now it's less than a dollar.
For me if the other providers will not drop their prices dramatically in the coming weeks I see no reason to use them. Even with a 200 dollar subscription, paying per token for DeepSeek is better value.
Unfortunately, it made a mistake and bricked it. I handed over to Astra but noticed that my watch was dead, it must have died immediately before. Too bad, unlucky me.
It’s funny, in oh-my-pi I’ve had much better luck with DS 4.1 Flash than GLM 5.3 flash. Glm seems to just constantly reason in circles before attempting a tool call
Are you using the API directly from Deepseek? Otherwise, performance will suffer greatly. If you're using OpenRouter make sure to pin Deepseek as the provider. If you're using OpenCode Go who knows who the provider is.
How are you holding it? You'll get wildly different real-world results from different inference providers, for one. For two there's the harness and what you do with it. In general you'll need to provide more direction to lightweight models, better guardrails, more planning, tighter goals.
I find this - or perhaps the title - a bit surprising.
I've benchmarked GLM 5.3 and DSv4.1-F on my fully-annotated decomp of the Nintendo 3DS's kernel, which I have a good mental understanding of, tasking them to find vulns and other bugs (in Max mode w/ subagents). GLM 5.3 founds almost all the vulns in 30min for $22, while DS only found one vuln for $2 in 40min.
Perhaps DS works better where targets have low-hanging fruits than can be found fast?
Also depends which glm 5.3 you use. I used open router and sometimes had good, then appalling performance. It turned out the provider being routed to - some were fp8 and some were fp4. The fp4 ones glitched on anything complex. It is such a difference that I only allow fp8 providers in my allow list.
Makes sense, GLM 5.1/5.2/5.3 is a lot better than DS v4.1, I found the same results in other domains.
Given how fast and cheap DS is, it's just an ideal model with enough "IQ" to let it loose. Another thing they left out of the article, DS becomes really good if you provide custom tools for the task, on it's own it's mediocre.
Worth saying that GLM-5.3 isn't GLM-5.3-Flash's "big brother" the way one might think. GLM-5.3-Flash is not GLM-5.3 scaled down. While GLM-5.3 is based on GLM-5.2, and "every gain comes from post-training" (https://z.ai/blog/glm-5.3), GLM-5.3-Flash uses a newly trained multimodal base model (https://z.ai/blog/glm-5.3-flash).
Both DS and GLM had the same numbers of subagents, 5 or so.
But, well, number of subagents doesn't make a difference if model is dumb (GPT 5.4 High, in May,in Chat mode outperformed what I see with DS4.1-F).
That being said, pricing model makes a huge difference for "find at least one" tasks: with API/PAYG if you have a chance to save 90%, you go for it, whereas with subscriptions it is optimal to burn all your remaining allowance right before reset
DS has perf issues if you parallelize it heavily (24+), especially when the context window is above the limit, on a single machine (with custom llm gateway): it fails 4x more often, and is 2x slower than gpt-5.6.
If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.
A little off-topic: where does one use those models such as GLM or DS for this kind of reverse engineering tasks? I think I read many of them refuse to help with tasks like those on their official platforms.
You can run deepseek-v4.1-flash on a local server with 4xH200@141GB for around 180K EUR. It can serve more than one user so it can be a good investment for a company, especially if you need your data to stay local (although I would advise do go a step up and populate with 8xH200 to serve many more users, ~300K EUR). You can run whatever test you want and you won't be limited by a front-end that pre-filters your requests.
They provide serverless access, and if you use fireconnect it's also easy to hook the models up to e.g. codex which allows for fairly clean comparisons
I think they provide the bare model (no filter) but I might be wrong, haven't tried using it for security research yet
It was very helpful in the aftermath of a client whose WordPress got stuffed up and analysing when it happened, how it happened, and what the entry point was. The American frontier models refused to help because, well, they just refuse to help with that kind of thing.
What I find surprising is that DS Flash can do it at all.
I love DS flash, it is an amazing workhorse to implement plans created by more robust models (such as GLM). But a more fair comparison would be of DS Flash with GLM Flash.
Thing is that GLM 5.3 is many multiples the cost to run, and slower.
I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond well to empirical evidence rather than go in circles.
So it might need some prodding, but it's likely in this case it was able to brute force after several runs and collecting some evidence.
How do I make deepseek "hack" my source code? do I just start my coding agent in my directory and command it to "find vulnerabilities", or is there some more sophisticated software to do that?
Start with an existing open source AI pentesting / source code scanning tool. Shannon and Strix are 2 that come to mind. They provide the tools, prompt, and frameworks for the model to use.
I'm a deepseek fanboy, but has anyone else found flash to not meet expectations? I've found it to be wildly bad at doing as asked, overengineering, and always assuming instead of reading even if told to read things in full before doing anything. It makes wildly silly mistakes in code and so far has been quite frustrating to work with. Maybe its just what i work on that its particularly bad at but in general feels like a strict regression on previous offerings.
Good luck building something of value with "day to day coding". If you're going to compete in the future economy, you're going to compete at the frontier. The low hanging fruit will be of no value. If I can recreate your SaaS with an open weight model at a fraction of the cost, then your SaaS has no value.
the vast majority of successful SaaS offer capabilities or services no where near the frontier.
furthermore i'd argue that the success of an saas, or really any startup, has more to do with things aside from the code and service itself.
Finance management, social connections and effects, word of mouth, networking -- probably all more important than whether or not the todo app uses a functional language and is formally proven.
What the hell are you yammering about. Everything in the economy exists at different quality/price tradeoffs, everything from tomato sauce at the grocery store to landscaping services to SaaS. "Compete at the frontier" is some maximalist buzzword nonsense. Do you think Ikea is crying because their furniture is cheap and they're not "competing at the frontier"?
we recently got this running in 192gb of vRAM and using it with the Klaudia harness has been incredible for driving out work that we'd need to use Opus and Fable for previously
When the history books are written and all is said and done, the hubris of this moment where all the American labs decided to punk their investors and join hand in hand in agreeing to let the Chinese win forever is going to be the main story.
> When the history books are written and all is said and done, the hubris of this moment where all the American labs decided to punk their investors and join hand in hand in agreeing to let the Chinese win forever is going to be the main story.
Come on. Lecturing about hubris when your message is damn the consequences, full speed ahead?
If it's a race to build the torment nexus, or a race with a nonzero chance of building the torment nexus by accident, I don't care about winning.
But it isn't a race? I feel like maybe we read different comments... It didn't seem they were saying the first was the winner, just that a closed system can never beat an open one.
Win what? The race to the bottom always has this competitive language.
“If we ban CFCs now the Chinese will win!”
“If we ban chemical weapons, nuclear weapons, etc etc our enemies will triumph! They won’t stop!”
“If we switch to biodegradeable plastic then our rivals will have an advantage.”
“If we dont externalize the costs to our population, then they will, and then will win!”
I think workflows can do the job agents do, 20x cheaper and more predictably and safely. They can completely displace agents, just as HFCs displaced CFCs and then we were able to ban CFCs and phase them out through international COOPERATION. The language of COOPERATION is what saves us vs COMPETITION is all about cutting corners and externalizing costs. Google the Montreal Protocol, Geneva Conventions, Nuclear Non Proliferation Treaty, Unleaded Gasoline etc etc.
Agents have got to be marginalized. They are just popular because the labs need to make a ton of money for their investors and recoup their massive spending on training models.
We're talking about replacing most uses of Agents with uses of Workflows. That's the key first step, without which the industry will cry "don't regulate us... do you want China to WIN???"
We ran v4.1 Flash through our evaluations and found it to be smarter and faster than V4 Flash, with a commensurate price bump. Some notes:
- Because it's a heavy reasoner, it sits near Gemini 3.7 Flash on the Pareto front (not as cheap as the price suggests in practice).
- Closer than expected to the top open weights models (GLM 5.3 and Kimi K3) in agentic coding, at lower cost.
- Chinese models have always been strong iterators in an agentic harness. This model is no different, reaching an average percentile ~20% higher when given a harness vs a one-shot solution. That one-shot fluid intelligence is what makes a model feel smart, though, and typically results in fewer attempts/tokens to solve a problem, and American frontier models are still far ahead in that department.
The new architecture is interesting. It puts pricing between their old Flash and Pro lineups, suggesting they might be abandoning their super-cheap flash models (which weren't that fast due to heavy reasoning) and their pro models (which sort of flopped and weren't consistently better than their flash models, despite the size/cost) and shipping a strong intermediate that competes with the Gemini Flash series.
Data at https://gertlabs.com/rankings
DeepSeek models have such good benchmark performance, amazing pricing, and the team over there seems to be widely considered impressive.
I just haven't found them to be very good? I've had a ton more success with the GLM models (since 5.2 anyway). Maybe I'm just holding it wrong, DS models seem to get stuck in loops or tell me nonsense. GLM feels like budget Claude.
My experience is complete opposite from yours. The V4 Flash was already quite good, but V4.1 is really very close to SOTA in my books. I've eval'd these models for weeks against Gemini 3.8, Kimi K3 and Opus 5, and DeepSeek absolutely wins these evals.
It's really crazy. We price per token our customers. If Kimi was about 30% of the price of Opus 5 for the same quality, DeepSeek is 1/10th of a price of Kimi K3. We've come down in price so much that I seriously cannot recommend other models before they reduce their pricing.
And what is really interesting is its programming ability. As I've said in my previous comments, I use agents a lot in my work. Since early Opus days until now I have 7-8 agents working in parallel for different tasks. Rust, design, GEPA, evals, analysis. For a long time Kimi K3 was the best model for this work, and before that GPT 5.5. But I still can't really believe how well DeepSeek works here. I really try to find faults from it, trying to see that it must be doing sloppy work and be worse than the others. But it does not. It finishes every task I give to it. And the cost per task is under a dollar, usually 15-30 cents.
In comparison the same task with Kimi would be 3-15 dollars; sometimes closing to 100. And before that with GPT 5.5 a 800 dollar task was not uncommon if I spent days evaluating models.
Now it's less than a dollar.
For me if the other providers will not drop their prices dramatically in the coming weeks I see no reason to use them. Even with a 200 dollar subscription, paying per token for DeepSeek is better value.
My harness: https://omp.sh/
DS 4.1 Flash did manage to hack my watch:
https://github.com/skorokithakis/amazfit-neo-hacks
Unfortunately, it made a mistake and bricked it. I handed over to Astra but noticed that my watch was dead, it must have died immediately before. Too bad, unlucky me.
Depends on a lot of factors, AFAIK Claude Code (TUI Harness) does not work well with Deepseek, but Opencode is surprisingly good.
What harness are you using? I've had incredible luck with OMP and Deepseek v4 Flash.
OMP and DS are a match made in heaven.
It’s funny, in oh-my-pi I’ve had much better luck with DS 4.1 Flash than GLM 5.3 flash. Glm seems to just constantly reason in circles before attempting a tool call
Are you using the API directly from Deepseek? Otherwise, performance will suffer greatly. If you're using OpenRouter make sure to pin Deepseek as the provider. If you're using OpenCode Go who knows who the provider is.
> DS models seem to get stuck in loops
I have never had looping issues with DeepSeek models. Which provider/serving framework and harness are you using?
Have you tried Flash V4.1?
How are you holding it? You'll get wildly different real-world results from different inference providers, for one. For two there's the harness and what you do with it. In general you'll need to provide more direction to lightweight models, better guardrails, more planning, tighter goals.
I find this - or perhaps the title - a bit surprising.
I've benchmarked GLM 5.3 and DSv4.1-F on my fully-annotated decomp of the Nintendo 3DS's kernel, which I have a good mental understanding of, tasking them to find vulns and other bugs (in Max mode w/ subagents). GLM 5.3 founds almost all the vulns in 30min for $22, while DS only found one vuln for $2 in 40min.
Perhaps DS works better where targets have low-hanging fruits than can be found fast?
Also depends which glm 5.3 you use. I used open router and sometimes had good, then appalling performance. It turned out the provider being routed to - some were fp8 and some were fp4. The fp4 ones glitched on anything complex. It is such a difference that I only allow fp8 providers in my allow list.
How do you know which providers are good?
In openrouter you can pin providers and you can filter by quantization.
Makes sense, GLM 5.1/5.2/5.3 is a lot better than DS v4.1, I found the same results in other domains.
Given how fast and cheap DS is, it's just an ideal model with enough "IQ" to let it loose. Another thing they left out of the article, DS becomes really good if you provide custom tools for the task, on it's own it's mediocre.
How does GLM 5.3 Flash rank against it's big brother and the latest Deepseek?
Worth saying that GLM-5.3 isn't GLM-5.3-Flash's "big brother" the way one might think. GLM-5.3-Flash is not GLM-5.3 scaled down. While GLM-5.3 is based on GLM-5.2, and "every gain comes from post-training" (https://z.ai/blog/glm-5.3), GLM-5.3-Flash uses a newly trained multimodal base model (https://z.ai/blog/glm-5.3-flash).
Both these models are not bad but you can not trust them to make good decisions. I meant GLM 5.1/5.2/5.3 in the previous answer, not the flash version
In your example, couldn't you parallelize DS's work more ? You could have 11 times as many agents for the same price.
Both DS and GLM had the same numbers of subagents, 5 or so.
But, well, number of subagents doesn't make a difference if model is dumb (GPT 5.4 High, in May,in Chat mode outperformed what I see with DS4.1-F).
That being said, pricing model makes a huge difference for "find at least one" tasks: with API/PAYG if you have a chance to save 90%, you go for it, whereas with subscriptions it is optimal to burn all your remaining allowance right before reset
DS has perf issues if you parallelize it heavily (24+), especially when the context window is above the limit, on a single machine (with custom llm gateway): it fails 4x more often, and is 2x slower than gpt-5.6.
If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.
> If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.
Inference providers can credibly promise to not train on your data if they are in a position to get sued.
A little off-topic: where does one use those models such as GLM or DS for this kind of reverse engineering tasks? I think I read many of them refuse to help with tasks like those on their official platforms.
You can run deepseek-v4.1-flash on a local server with 4xH200@141GB for around 180K EUR. It can serve more than one user so it can be a good investment for a company, especially if you need your data to stay local (although I would advise do go a step up and populate with 8xH200 to serve many more users, ~300K EUR). You can run whatever test you want and you won't be limited by a front-end that pre-filters your requests.
I've been testing these models with fireworks.ai
They provide serverless access, and if you use fireconnect it's also easy to hook the models up to e.g. codex which allows for fairly clean comparisons
I think they provide the bare model (no filter) but I might be wrong, haven't tried using it for security research yet
I use DS straight off of DeepSeek's API.
It was very helpful in the aftermath of a client whose WordPress got stuffed up and analysing when it happened, how it happened, and what the entry point was. The American frontier models refused to help because, well, they just refuse to help with that kind of thing.
GLM-5.3; well, the flash variant at least; you just ask what to do and it complies with no objections :^]
GLM 5.3 doesn't seem to refuse vuln research (which it classifies as "audit") and is good at it.
Therefore you use for offensive cybersecurity tasks because Daybreak Red/Mythos is pure unobtainium for us mere plebians.
DB Blue thankfully exists, but I suspect you risk a ban if you use it with codebases you neither own nor use
Tl;dr because it's the only model at the level of 5.4~5.6 that doesn't refuse tasks nor risk your oai account getting banned
For plain RE tasks Sol or Astra should work just fine (I think)
I think it really depends on what the data DS was fine tuned on. If your use case is very specific, it wouldn’t have distilled that knowledge well.
you can run 10 ds-f agents at the same time for the same cost exploring a much wider surface area.
Which harness were you using?
What I find surprising is that DS Flash can do it at all.
I love DS flash, it is an amazing workhorse to implement plans created by more robust models (such as GLM). But a more fair comparison would be of DS Flash with GLM Flash.
Thing is that GLM 5.3 is many multiples the cost to run, and slower.
I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond well to empirical evidence rather than go in circles.
So it might need some prodding, but it's likely in this case it was able to brute force after several runs and collecting some evidence.
Seems pretty bold to claim deepseek is the "best hacking model" while providing zero comparisons to other models...
Notice the qualifier "our", that is the one they have access to.
What if there would be a separate category for distilled models?
How do I make deepseek "hack" my source code? do I just start my coding agent in my directory and command it to "find vulnerabilities", or is there some more sophisticated software to do that?
Start with an existing open source AI pentesting / source code scanning tool. Shannon and Strix are 2 that come to mind. They provide the tools, prompt, and frameworks for the model to use.
https://github.com/KeygraphHQ/shannon https://github.com/usestrix/strix
As opposed to what? Is enclave.ai signed up for GPT Cyber or Glasswing?
Interesting result, but the writing is very poor.
I'm a deepseek fanboy, but has anyone else found flash to not meet expectations? I've found it to be wildly bad at doing as asked, overengineering, and always assuming instead of reading even if told to read things in full before doing anything. It makes wildly silly mistakes in code and so far has been quite frustrating to work with. Maybe its just what i work on that its particularly bad at but in general feels like a strict regression on previous offerings.
DeepSeek is underrated. Basically all Chinese models are good enough for day to day coding at this point.
The 2 trillion dollar ROI on anthropic alone?
Good luck with that.
Good luck building something of value with "day to day coding". If you're going to compete in the future economy, you're going to compete at the frontier. The low hanging fruit will be of no value. If I can recreate your SaaS with an open weight model at a fraction of the cost, then your SaaS has no value.
the vast majority of successful SaaS offer capabilities or services no where near the frontier.
furthermore i'd argue that the success of an saas, or really any startup, has more to do with things aside from the code and service itself.
Finance management, social connections and effects, word of mouth, networking -- probably all more important than whether or not the todo app uses a functional language and is formally proven.
What the hell are you yammering about. Everything in the economy exists at different quality/price tradeoffs, everything from tomato sauce at the grocery store to landscaping services to SaaS. "Compete at the frontier" is some maximalist buzzword nonsense. Do you think Ikea is crying because their furniture is cheap and they're not "competing at the frontier"?
If I can recreate your "frontier" SaaS with day-to-day coding, it has no value. Or perhaps the few months' lead was never the deciding factor.
Sounds like you suffer from Dunning-Kruger. AI is making a lot of dumb people look smart. A lot of us can see right through it.
we recently got this running in 192gb of vRAM and using it with the Klaudia harness has been incredible for driving out work that we'd need to use Opus and Fable for previously
Edit: Comment deleted - no longer useful.
Apparently they updated it perhaps based on your comment?
> . The accepted runs cost $4.65. Failed attempts and replacement runs increased the complete cost to $5.14.
Oh - I see that now. It's possible I missed it originally - I did read the article but I was skimming quickly. Mea culpa, if so.
You can have 100 runs for the price of the Claude Max plan?
Or the price of a GPT Pro 20X plan... er, wait, except you can't go sign up for one of those right now.
When the history books are written and all is said and done, the hubris of this moment where all the American labs decided to punk their investors and join hand in hand in agreeing to let the Chinese win forever is going to be the main story.
> When the history books are written and all is said and done, the hubris of this moment where all the American labs decided to punk their investors and join hand in hand in agreeing to let the Chinese win forever is going to be the main story.
Come on. Lecturing about hubris when your message is damn the consequences, full speed ahead?
If it's a race to build the torment nexus, or a race with a nonzero chance of building the torment nexus by accident, I don't care about winning.
> Lecturing about hubris when your message is damn the consequences, full speed ahead
They said that... where?
> They said that... where?
By dismissing concerns and focusing exclusively competition and "winning the race."
There's a race? I didn't see that at all in the OP
>in agreeing to let the Chinese win forever
Seems like a race to me.
But it isn't a race? I feel like maybe we read different comments... It didn't seem they were saying the first was the winner, just that a closed system can never beat an open one.
Win what? The race to the bottom always has this competitive language.
“If we ban CFCs now the Chinese will win!”
“If we ban chemical weapons, nuclear weapons, etc etc our enemies will triumph! They won’t stop!”
“If we switch to biodegradeable plastic then our rivals will have an advantage.”
“If we dont externalize the costs to our population, then they will, and then will win!”
I think workflows can do the job agents do, 20x cheaper and more predictably and safely. They can completely displace agents, just as HFCs displaced CFCs and then we were able to ban CFCs and phase them out through international COOPERATION. The language of COOPERATION is what saves us vs COMPETITION is all about cutting corners and externalizing costs. Google the Montreal Protocol, Geneva Conventions, Nuclear Non Proliferation Treaty, Unleaded Gasoline etc etc.
Agents have got to be marginalized. They are just popular because the labs need to make a ton of money for their investors and recoup their massive spending on training models.
Those comparisons are pretty irrelevant
You can't compare banning football to banning genetic experiments and say "they are both bans and therefore directly comparable"
No one was talking about football.
We're talking about replacing most uses of Agents with uses of Workflows. That's the key first step, without which the industry will cry "don't regulate us... do you want China to WIN???"