i would love a deterministic program that can confidently make plans for lower-cost models like deepseek. ofcourse the LLM part wont be deterministic, but its a lot easier to measure quality like this. you could argue an AGENTS.md is this, but from experience its not enough to make non-frontiers act have a high success rate.
I've been working on a pi extension to do this, after frustration with getting the best current local models to stay on track, and just to deal with their relative slowness; gist is that it uses pi hooks to keep the model deterministically on track and beads-rust issue tracking tool to keep everything organized.
A SOTA model writes the initial prompt, and creates the beads issues; then a continuous iteration of plan(local) -> review (sota) -> implement(local) -> review (sota). Until the sota reviewer model is happy with the implementation.
And can mostly just let-it-run; e.g. overnight since local models with mac unified memory are slow.
Still early days, but have had reasonably good success with a Defender (1981) clone and now I'm having it work on a Prince of Persia clone (both Go/ebiten).
Note: I think this is only possible now because Qwen3.8-Flash-Next and 27b are incredibly good models.
I’ve had success with writing eg shell scripts that have a deterministic scaffold for the thing I’m actually trying to do, then call off to the agent for only the things that actually need it. For example I was trying to find a race condition flaky bug in my code, so the shell for loop ran the build N times, and called out to the agent to analyze the build logs if the tests failed, then the shell would ping me on slack when it was done, so the whole thing could run in the background.
I’ve been thinking more about how this deterministic + agents style could work, it’s kinda like the analogy of factories in the 1800s going from the central shaft to electric motors on each desk (where the central shaft in this case is a chat window, and the motor on a desk is calling the llm api from wherever you want just like a normal api call)
Of course, maybe in a few months the agents would just be reliable enough to do the shell part on their own too, but we’ll just have to wait and see
I like your analogy. The main problem though is context and keeping it clean as much as possible as long a parallelization. This is what drove to build this tool: having control of everything that the LLMs will do, controlling all with one main planner that orchestrates the rest. This way we can have cheaper LLMs with a short context window used (less intelligence degradation) while still obtaining the same objective.
And again, you can have a clear picture of everything structured as tasks.
until we can get to rely on huge swarms of agents (tasks) being directed on the planner alone I don't see how we can get a better framework.
There are a lot of ways to slice the problem of getting the agents to complete a goal without getting lost, and the right solutions are somewhat problem-specific. For the projects I've done 35B Qwen is about the smallest that seems to make useful progress in a general purpose harness while 4B Qwen is workable with a task-specific harness. At the lower end the plan has to be traditional search/planner techniques in code not something the model has any control over, of course that limits the kinds of problems that fit. The high end coding models are perfectly capable of making a functional 1-off harness for those jobs so it ends up not being that bad to implement.
Yeah it's exactly what I've seen as well. Ordewell is close to your high end case: a frontier model builds the plan once, then it's fixed, the executing model can't reinterpret it.
Won't get you to 4B, but should help a small model that only has to execute, not plan and execute at once.
I thought about it but where the tool shines are large undefined tasks, which are complex to quantify and test (not as easy as implementing a simple bug fix that you can test directly). Even if I were to find such test dataset it would probably require a lot of money to reach statistically significant results.
Either way, the framework is inspired a lot from Matt Pocock, and is pretty much established.
Fair — AGENTS.md is prose the model has to re-interpret every session, and that reinterpretation is exactly where weaker models lose the thread. Here the plan is parsed and enforced as structured data: tasks with declared dependencies and one prompt each, so the per-step job is smaller and the plan isn't up for renegotiation. Nothing in that needs a frontier model
I just haven't benchmarked it against deepseek-class runners, and the runner is pluggable if you want to be the one who does.
To me that's not the embarrassing part... unmaintainable would be. Using AI for coding and writing doc is standard practice now, not using it would be insane.
i would love a deterministic program that can confidently make plans for lower-cost models like deepseek. ofcourse the LLM part wont be deterministic, but its a lot easier to measure quality like this. you could argue an AGENTS.md is this, but from experience its not enough to make non-frontiers act have a high success rate.
Have you looked at AWS's AI-DLC setup?
https://github.com/awslabs/aidlc-workflows
very interesting, the concept is the same. I see they flagged Opus as the suggested planner, which makes a lot of sense. Thanks
I've been working on a pi extension to do this, after frustration with getting the best current local models to stay on track, and just to deal with their relative slowness; gist is that it uses pi hooks to keep the model deterministically on track and beads-rust issue tracking tool to keep everything organized.
A SOTA model writes the initial prompt, and creates the beads issues; then a continuous iteration of plan(local) -> review (sota) -> implement(local) -> review (sota). Until the sota reviewer model is happy with the implementation.
And can mostly just let-it-run; e.g. overnight since local models with mac unified memory are slow.
Still early days, but have had reasonably good success with a Defender (1981) clone and now I'm having it work on a Prince of Persia clone (both Go/ebiten).
Note: I think this is only possible now because Qwen3.8-Flash-Next and 27b are incredibly good models.
- https://github.com/Dicklesworthstone/beads_rust
I’ve had success with writing eg shell scripts that have a deterministic scaffold for the thing I’m actually trying to do, then call off to the agent for only the things that actually need it. For example I was trying to find a race condition flaky bug in my code, so the shell for loop ran the build N times, and called out to the agent to analyze the build logs if the tests failed, then the shell would ping me on slack when it was done, so the whole thing could run in the background.
I’ve been thinking more about how this deterministic + agents style could work, it’s kinda like the analogy of factories in the 1800s going from the central shaft to electric motors on each desk (where the central shaft in this case is a chat window, and the motor on a desk is calling the llm api from wherever you want just like a normal api call)
Of course, maybe in a few months the agents would just be reliable enough to do the shell part on their own too, but we’ll just have to wait and see
I like your analogy. The main problem though is context and keeping it clean as much as possible as long a parallelization. This is what drove to build this tool: having control of everything that the LLMs will do, controlling all with one main planner that orchestrates the rest. This way we can have cheaper LLMs with a short context window used (less intelligence degradation) while still obtaining the same objective. And again, you can have a clear picture of everything structured as tasks. until we can get to rely on huge swarms of agents (tasks) being directed on the planner alone I don't see how we can get a better framework.
There are a lot of ways to slice the problem of getting the agents to complete a goal without getting lost, and the right solutions are somewhat problem-specific. For the projects I've done 35B Qwen is about the smallest that seems to make useful progress in a general purpose harness while 4B Qwen is workable with a task-specific harness. At the lower end the plan has to be traditional search/planner techniques in code not something the model has any control over, of course that limits the kinds of problems that fit. The high end coding models are perfectly capable of making a functional 1-off harness for those jobs so it ends up not being that bad to implement.
Yeah it's exactly what I've seen as well. Ordewell is close to your high end case: a frontier model builds the plan once, then it's fixed, the executing model can't reinterpret it. Won't get you to 4B, but should help a small model that only has to execute, not plan and execute at once.
Have you quantified the performance on any particular benchmark?
I thought about it but where the tool shines are large undefined tasks, which are complex to quantify and test (not as easy as implementing a simple bug fix that you can test directly). Even if I were to find such test dataset it would probably require a lot of money to reach statistically significant results. Either way, the framework is inspired a lot from Matt Pocock, and is pretty much established.
What sort of determinism do you have in mind?
Fair — AGENTS.md is prose the model has to re-interpret every session, and that reinterpretation is exactly where weaker models lose the thread. Here the plan is parsed and enforced as structured data: tasks with declared dependencies and one prompt each, so the per-step job is smaller and the plan isn't up for renegotiation. Nothing in that needs a frontier model I just haven't benchmarked it against deepseek-class runners, and the runner is pluggable if you want to be the one who does.
I opened the page looking for a differenciator, but everything about the project is AI-written (including author replies in these comments).
To me that's not the embarrassing part... unmaintainable would be. Using AI for coding and writing doc is standard practice now, not using it would be insane.
Fair, and I won't pretend otherwise. I heavily used AI to help write the docs and code. The thing is though... I designed it and I stand behind it.
But why write these comments with AI?
Complete brainrot lol
so why did it get upvotes and a place on coveted frontpage?
have you tried https://github.com/highflame-ai/codeoid
nope but I will check it out, thanks