> The rocket launch was only achieved through significant ‘manual’ effort on Sol’s part as it papered over its late-game failures to connect certain production systems by repeatedly ferrying supplies between areas of the factory.
Hah I dunno, I think most human players wouldn't find that very fun and would instead take that as an opportunity to refactor the factory for optimized performance (i.e. the core gameplay loop)
When you just need a few parts to be made, it is sometimes very tempting to drop a chest, fill it up with parts, then go do something else, one of the 1000 tasks you have been neglecting.
That's definitely true, but only to a certain point, after a while it becomes too monotonous to dig around in chests building and combining stuff in an ad hoc manner. LLMs have no problem with that kind of repetitive minutia though.
Logistic bots, once they are unlocked, the game becomes trivial.
Once you make a few blueprints, then the real joy of factorio is found. Nothing like splatting down an outpost and watching it sprint into existence as thousands of “agents” build it out.
my husband plays exactly like that, and whines until I log onto the multiplayer server so that I can "make it work like a programmer".
I then log in, create a bunch of logistics chains, and then a few days later he whines about it again when he enters the next phase of gameplay.
This continues until fast logistics drones, then he takes over; less need to consider routing & order .
Factorio is interesting, a lot of players just treat it like they're playing Postal and just want to gun bugs down in creative mode equivalents, meanwhile i'm fretting about green boards per second.
Yes and no - for a one-off I will just craft something "by hand" instead of doing a ground-up rebuild. (The exception, of course, was my build for the "Lazy Bastard" achievement.) I don't go into it thinking "oh boy it'll be fun to mule around these items across the width of my factory!" but it's also not so offensive that I strictly automate everything all the time. Not that I play the game entirely the way it's meant to be played
Naturally, "I'll do it once" starts as a temporary crutch and ends up a load-bearing liability. A phenomenon entirely absent from our day jobs ;)
When you start a new game in Factorio, if you never played it, you can’t know how much of each resource you’ll have to extract and neither how much of an item you’ll need to produce.
I know it’s easy to forgot after thousands hours of playtime, but the game starts with your character having to manually mine coal one by one.
> When you start a new game in Factorio, if you never played it, you can’t know how much of each resource you’ll have to extract and neither how much of an item you’ll need to produce.
Some of this information should be recoverable somewhere in the tech tree tooltips, if nowhere else.
More importantly, vanilla Factorio plays a nasty little trick on first-time players: The final sciences and launching the rocket both require scaling up. This means that either you refactor your factory, or you grind out that first launch super slowly using an inadequate production line.
Veteran players know ways to design some modularity into a factory, and when to yank out subsystems and rebuild them bigger somewhere else.
Factorio: Space Age is actually more forgiving about scalability. You don't need to scale up as much, and you have much more powerful tools for doing it. Instead, Space Age has lots of novel game mechanics to master, and higher total system complexity across the solar system.
Not true, every item shows its raw materials right there in its menu. The rest is just arithmetic.
Also the LLM probably has seen dozens of factorio-related websites and setups.
And also, player (artificial or natural intelligence) don't need to setup everything upfront, it's actually the main gameplay feature is to run between production areas to expand/optimize them.
Technically every item displays requirements and crafting time, but most human players will just eyeball production rates until they start trying to build larger scale factories that you can't easily jerry-rig fixes onto.
Possibly there is a lack of item-throughput figures for belts and stuff though.
A bus doesn't solve this, because it still is contained by items/second. You can't just select copper, if there's no more copper on the belt. And you might not be able to jerry-rig another belt of copper onto your bus.
A bus doesn't solve the issue the person you replied to was talking about.
IME the models know some basic information about all the popular games, including some basic strategy, but they don't know/remember the details to be even a moderately familiar player. And usually if you ask a question they will google it.
Yet, that's not how new players (human or AI) approach the game. If you look at AlphaZero, probably state of the art in learning e.g. chess (more information about rules than Factorio has in game) from scratch, using only rules, it just plays a lot of games with itself. It doesn't "just think a lot" to come up with a perfect strategy; it experiments.
The AI has trained on the entire factorio wiki along with every tutorial video ever made and who knows what else. I don’t see room for hidden anything in that scenario.
I am not the OP but I have (many) hours in Factorio and I think what they might be talking about is things like how it’s not apparent at the start just how much copper infrastructure you will need by the time you get to rockets and space platforms because the early recipes are iron-heavy. Or how important green circuit scaling will become. Or how much oil cracking might change your layout.
If you know what’s coming you make different choices in how you build that heads off a fair bit of refactoring.
If you haven't been keeping an eye on LLMs playing games, GPT 6.1 Sol can new beat games 100% vision-only (without harnesses that give the agent extra information about the game). Pokemon is no longer a difficult challenge for it, even with additional difficulty restrictions such as hard mode and level caps. Recently, GPT 6.1 Sol beat Final Fantasy VII and its bonus bosses vision-only surprisingly quickly.
I suspect Opus 5.5 can do even better but there aren't as many people testing it.
Where are the recordings of these games? I know this kind of stuff must be happening, but it seems oddly difficult to track down the actual recordings, which feel like the really fascinating thing people should be publishing if they’re putting in the time and tokens to do stuff like this.
Like I want to see what Pokemon strategies it uses: Does it try to brute force with its starter? Does it try to load all of its HMs onto one or two Pokemon? Does it use Flash? If it needs to catch a new Pokemon to beat a gym leader, how much does it remember about which species it encountered where and what their levels were?
The LLMs no longer brute force a starter, and they have to play competently, including the use of setup. Here's a screenshot of the completion of the GPT-6 Astra run in 21 hours with comically underleveled Pokemon (and it actually savescummed RNG!): https://www.reddit.com/r/ClaudePlaysPokemon/comments/1w79vim...
It has been a few years now. By now you should have realized that AI is for doing stuff at a massive x100 productivity boost and never getting a complete project done. (For some reason there's those "final kinks to iron out", once the AI figures out where it fucked up. Just two more prompt modifications.)
You could make the same argument for these LLMs years ago but there's a reason that they've become faster and more accurate at beating these games. GPT 5.5 took 218 hours to beat Pokemon Fire Red; Astra took 21.
Only months ago did these LLMs make it past the Victory Road boulder puzzle without harness assistance.
I would be willing to bet it can beat games un-released at training time as long as latency is not a requirement, which does limit the game types a bit.
How do they feed the game screen to the models? I imagine the models are too slow if you send 60FPS, but I feel like that’s also the case if you maybe feed it 1FPS? If you go any slower, you’ll miss too much info?
Essentially yes. For games with some real-time component (FFVII), the game pauses after actions are executed to take a screenshot and let the agent think.
Opus and claude models have always been lagards in vision. It just hasn't been a focus like coding for anygropic. Gemini and ChatGPT have always been much stronger in this area
I would like to see it play Slay the Spire (or 2) and beet it consistently. Mainly to see what strats it uses. I suspect it wouldn't have too much trouble.
Not sure about LLMs but I yearn for someone to train neural nets to play the NPCs in a shooter game well but without having aimbots or seeing through walls - basically a “true” version of F.E.A.R. AI where they play simulated squads that overwhelm the player with various tactics and adapt to their playstyle. Or even just make increasingly complex state machines and goal oriented systems which might be more performant but still encode the main rules.
You want to create a human feeling using AI, but AI is not needed for the feelings you want to create. All you really want is a feeling of variety and surprise in the way computer opponents behave. A LLM-level of AI might just do the same thing all the time and become quite boring.
And yet, that argument I just made often bothers me, because I do think there is a place for strong AI in games, and many people just reflexively say "strong AI wouldn't be fun". I think they are partly right, but I also think a strong AI that is purposely dumbed down in fun ways could be really enjoyable.
When it comes to shooters though, those games aren't really about competition and decisions, they are about giving the human player an experience. I do think strong AI is overkill in such situations.
A strong AI for chess absolutely improves chess computer games.
A strong AI for a Magic: The Gathering style card battler would improve the game.
A strong AI for Civilization would improve the game, but I think we're starting to get into a realm where you can fake some things. I do think Civilization games are lacking because of poor AI, but I don't know that we need LLM-levels of AI in Civilization.
Again, in shooters, there's no decision, no illusion of fair play or equal competition based on making decisions, it's all about the human experience, and so LLM-level AIs aren't needed here at all.
> You want to create a human feeling using AI, but AI is not needed for the feelings you want to create.
> When it comes to shooters though, those games aren't really about competition and decisions, they are about giving the human player an experience. I do think strong AI is overkill in such situations.
> Again, in shooters, there's no decision, no illusion of fair play or equal competition based on making decisions, it's all about the human experience, and so LLM-level AIs aren't needed here at all.
No, playing against humans absolutely sucks to an untold level for me - because more often than not at some point it ends up being about who has better aim or reaction times and I lose like most of those.
Stuff like that makes everything from CS, Valorant, Battlefield, Wardogs, Active Matter, Tarkov, ABI, Insurgency Sandstorm and any other shooter game unplayable. Meanwhile purely skulking around corners is no fun and also sucks, because that still doesn't resolve the imbalance, and that's after over a thousand hours and sometimes with SBMM and all. Teamwork can't really fix that, even when I tried playing a healer in Valorant friendlies would get taken out, I'd get picked off, round over, we lose.
What I really want is AI that is worse at aiming than I am but is more numerous and makes up for it with numbers and utilizing those numbers well with tactics. Games like FEAR and Trepang2 and Incursion Red River evoke that feeling, but could definitely be made even better in that regard! It would be strong not because of sniping me from across the map like a lazily made bot, but rather because there'd be grenades going out in my direction, they'd use cover well and suppress me while some others push from the flank, not just rush me with godlike aim.
That is before you consider playing online being a minefield (especially in Eastern European servers), it's a coin flip whether someone in Wardogs will be trying to flip choppers or AFK bot at base, or whether they'll decide that being annoying and sharing the best slurs they know is what they want to do today. It seems like there's a narrow band between tryhards who berate you for not playing good enough and people who are just trolling. I don't want that kind of hit or miss experience, ever. It's even the same with Arma RP servers (like Altis Life, the idea is cool) when I tried those - you'd be able to make money by mining and processing ores or whatever for a bit, until someone would randomly shoot you for no real reason.
I have been playing with this. Having tiny models provide more contextual information when an NPC speaks is cool but the "small enough" models tend to go off the rails.
It's fun if you have a handful of people playing, but it would quickly go towards negative reviews if you tried to sell it.
I don't think an LLM should ever be having a two-way conversation with the player. It would be more suited to 2 npcs having conversations amongst themselves. All dialogue delivered to the player should be human written.
At most it should be mildly tweaking existing dialogue, e.g updating a line to reflect game events. But as soon as you present a prompt window to the player the entire illusion that you are in a game world instantly dies.
> I don't think an LLM should ever be having a two-way conversation with the player. It would be more suited to 2 npcs having conversations amongst themselves. All dialogue delivered to the player should be human written.
I don't agree. All -important- dialogue yes. Background NPC chatter? Even when studios spend a lot of effort crafting multiple lines, pretty soon you have seen them all. They also don't change based on player actions.
i think it can work for procedurally generated situations, and the llm is acting more like a translator between data model and player rather than being a data storage layer.
thats close to tweaking existing dialog, but with a bit more freedom.
you still dont have to give the player arbitrary response
You absolutely cannot let the player get any whiff of AI slop so the integration needs to be extremely delicate. Watching a film and noticing a person has 6 fingers in a scene would instantly kill any engagement someone would have with it. For it to remain as art it must always be fundamentally about human-human communication, I want to have a conversation with the game developers through the game, not with a LLM. If developer intent ever becomes muddled the game has failed.
Most dialogue in games is crap. I would much rather have an LLM writing the dialogue so that it can include references to player choices over what currently exists.
I built a factorio "sidecar" LLM harness that would look at production stats every few minutes and identify issues. Even sonnet was good enough at finding potential things to alert on. coal production dropped and then iron production dropped two minutes later --> top potential issue.
Not useful enough that I actually play with it, but it was capable enough to be useful even just with sonnet.
It's a funny situation at the moment because most attempts are just sort of shoehorning LLM chat into a game that doesn't really need it, and then gamers quite rightly mock the results which don't really add anything to the game.
I think there's a bit of a lack of knowledge of game history on both sides, because we already know it can be fun when a game is designed around it. It worked in Façade (2005!) and it worked in Starship Titanic (1998!!), and both of those could have worked even better with the technology we have today.
One type of game that seems like an obvious one to me, is something a bit like The Sims except that the characters' decisions and speech are LLM-controlled. Maybe one of the characters is player-controlled, so you can interact and talk with everyone else. Or to describe it from the other direction, it'd be a kind of interactive fiction like AI Dungeon invented, but grounded in a physically simulated setting.
I sometimes wonder if Façade actually used neural networks for text generation or just simple markov chains... I remember pewdiepie saying melon and getting kicked out lol. Also the Sims would be awesome but imagine an RPG with rich worldbulding, and LLM's interacting with the lore, like in an Elder Scrolls or Fallout title. Wouldn't it be mind blowing?
There has to be a AAAA studio building the new, greatest MMO, that also utilizes LLMs as part of the game loop, surely?
(Not sure how prohibitive it would be to do anything interesting with local LLMs of any kind in a big game, given how many platforms/setups you have to make the stuff run smoothly on, and the sheer amount of RAM they would bind)
https://proxiai.com/ Will Wright (The Sims, etc) is dabbling with AI in his latest game. No idea if it'll be any good, but there's def companies out there trying.
AAAA studios don't take risks. Financing a top game is way too expensive to gamble on. Only indies and small studios working on cheaper games can risk trying unproven ideas.
not taking risks is the biggest risk in the times when the most anticipated game drop (gta 6) is probably going to be decompiled within days if not hours of release
With the aid of SOTA LLMs, you can encode specialist intelligence into computer programs without going through the roundabout route of training small LLMs. And even when you do go the route of ML, it doesn't necessarily have to be an LLM or even a Neural network.
I hope to experiment with using classic tabular reinforcement learning in game AI one day. Imagine a framework that makes setting up small Q-learning tables easy, and you can just sprinkle dozens of Q-learning algorithms throughout your traditional AI scripting.
These old basic algorithms often have mathematical proofs that they will converge to optimal behavior, and they do so fairly quickly. Turning the complex world of the game into a Q-table would require some creativity and judgement, but is not that hard I think.
In the end we would have a controllable AI that actually does adapt. That's my hope anyway.
This... Hopefully inference can become so cheap games can have simple LLM's integrated into them. Imagine an RPG with LLM's and with rich worldbuilding; like Fallout NV with LLM's. Honestly, scary!
Fable seems to be a potential franchise to do all this; imagine a "living fantasy world" where everything truly unfolds based on your decisions, giving NPCs all the functions/actions that a player has.
i think hooking in an llm to dwarf fortress for turning the very mechanistic descriptions into prose. same with the adventure mode talk. instead of a constant "well weve got beasts" an llm could read a bunch of the language tree to make a unique introduction for each.
same with the df artworks. get kitfox to design some art styles for dwarves, elves, goblins, and humans, and then instead of just reading
"this is a fine engraving of a dwarf. it concerns the events of 102 when ulrist fastbuttons was elected mayor. ulrist fastbuttons is sobbing"
LLMs could be interesting geopolitical roleplayers if instructed to faithfully assume the leadership traits or cultural biases of the their intended role, especially in games with expressive diplomacy systems like Stellaris or Europa Universalis. It could also make the diplomacy experience a lot richer because the LLM is able to interpret ligusitic tone and other verbal cues. The modern diplomacy experience is basically just a dialog tree with stats and dice rolls attached to it.
Unfortunately for it to be sustainable it would need to be an LLM with VRAM usage probably under a couple gigs max so it can run locally on the gaming PC. That said it may well be better than current AI in strategy games.
Yes, the insane hardware requirements make it totally impractical - that's why I say "turn based" because turn based games would at least allow for a practical gameplay experience even with remote LLM APIs.
I bet a small 1B model trained solely on how to play the strategy game would do better than current AI solutions for these games and that would probably be able to run even on integrated GPUs.
with better being more convincing, rather than just stronger
its the same for like a chess bot. i dont want a perfect stockfish that occasionally makes a serious blunder when a dice rolls a certain way. i want one that actually falls into traps
You could play against an AI right now on the Brood War ladder right now. Granted, you have to make it to S rank first because that's where the bots end up.
impossible, unfortunately, because LLMs suck at creative writing. without handholding, they are incapable of producing anything but cliche-ridden purple prose for young adults. you can feed it the first half of a book with lots of sex, violence, and profanity, ask it to continue the story, and it will degrade into safe slop within two chapters.
both American and Chinese labs aim their models at coding and knowledge work, so they penalize adult themes in post-training.
One limitation of these type of demos (and similar with Minecraft, etc) is that:
1. Putting into "peaceful" mode makes response time a non-factor
2. These models have read and have access to every guide and video ever published on the subject. Most humans, especially on first play, learn game systems and interactions from scratch, then develop novel strategies and plans and "hypothesis test" their way into beating the game.
This is still impressive compared to where we were a few years ago, and being able to synthesize all existing information about a game and then play it to completion in a semi-realistic setting is great.
At a quick glance someone might mistake this for "brilliant AI solves Factorio", when in reality, given a similar but different game where it wasn't gifted all the information or training data (e.g. massive videos and screenshots) it might fare terribly (as others have pointed out in comments).
Isn’t the point of arc agi 3 to measure general game intelligence even if it’s not a AAA game? Frontier models have obliterated that. And they do keep most of it private so no gaming there.
The models these days have 3 components that is for sure going to get better and better
1. Very good vision (it can identify almost anything in a picture)
2. Keyboard and mouse actions (slow but it does it well)
3. General world understanding (this one is toss up but given Astra or Fable, it surely does well even if it takes a while)
Now combine all these with specific RL environments that may use specific games which also generalizes to an extent and we have a system that can only get significantly better from here.
As for speed, I’d say it’s a question of how much money you are willing to throw (like Cerebras for example).
It doesn't seem to understand the core mechanic of Factorio, which is putting outputs onto belts and routing them to other inputs. It does this in a few places, but for the most part everything is put into chests and moved around by the player.
Handfeeding only has higher throughput for single trips/a single item type, definitely not for an entire factory. The player inventory and movement speed can't keep hundreds of assemblers working continuously. A high throughput starter factory looks something like this:
If you measure by the product of throughput and distance traveled, for an item with a stack size of 100 (e.g. metal plates), the player's inventory adds up to roughly the same capacity as 2500 yellow belts. And that's assuming 100% utilization running back and forth in both directions, with no unused inventory space and no wasted time.
A basic assembler can manufacture 2500 belts in about 21 minutes. So as a rule of thumb, if you're playing for longer than 21 minutes, belts are worth the investment (at least for bulk transport of raw materials).
(I guess maybe add an additional factor of at most √2, since the player can run diagonally and belts are orthogonal.)
Well, the 1 hour TAS speedrun is probably close to optimal and it also does lot of stuff that would be annoying to do in real human gameplay: https://www.youtube.com/watch?v=tImNejO5g48
If you don't have enemies, you have unlimited time. Doing it thru player is what it learned and wasn't pressured to do better (it has no fatigue as opposed to humans).
It's just easier to do manual stuff when the complexity of scaling the automation becomes cognitively demanding. A new player does mostly everything manually while experienced players have a natural intuition for planning and scaling the automation.
The biggest thing the bugs introduce from an LLMs perspective is a necessity for realtime response. Without them, the LLM can take as long as it wants to decide and do its next operation on the factory. With them, not responding fast enough results in death (particularly early game where a lot of biter combat is player driven). I presume it doesn't do great at this.
That was exactly my thought when reading the headline, before even looking into the article. Sure it's still somewhat complex to get to a rocket launch, but without the biters where is the challenge, other than complexity? It's like a big excel sheet with conveyor belts, pipes and electricity poles at that point only.
The more interesting challenge is the one you eliminated, complexity, in getting efficient logistics and arrangements. Dyson Sphere Program (until late in its existence) and Satisfactory don't have waves of enemies attacking your bases but remain challenging and a satisfying puzzle. I remember I used a constraint solver to help me optimize some late game recipes for Satisfactory.
Patience, for one. I reach a point where I'm looking at a mess of coal-fed sorters that need a complete rip-and-replace with electric ones, and that's usually the point where I close the game and go back to something easier like Kenshi.
I know it's a pipe dream, but these kinds of games (Factorio, Satisfactory, etc) would all benefit from sort of brain-game interface where I could just think about what I wanted to do.
The tedium and repetition is really what kills the enjoyment for me. The problem solving, planning, forethought, etc are super fun but it's killed by the former.
Blueprints and construction bots, plus the well-thought-out and extensively battle-tested set of keyboard commands for them, are basically a brain-game interface for Factorio on the level that Vim is for text. For the grandparent’s “rip and replace coal burners” I am hitting mod-D and dragging a selection box over the offending area, then hover-Q on a nearby element I want to use and shift-click to plan it, once I have planned one instance of a subpattern its mod-C select, click hold and walk in a direction to tile it in one dimension, mod-C select, click hold walk to extend this over the other direction. It is now done, the bots will handle everything else - because Wube are an incredible bunch of UX engineers, I can place the new building before the old buildings are even removed, leave, and trust that everything will be done correctly. Highly recommend getting a “Starter Bots” mod that feels appropriately balanced.
> I am hitting mod-D and dragging a selection box over the offending area, then hover-Q on a nearby element I want to use and shift-click to plan it, once I have planned one instance of a subpattern its mod-C select, click hold and walk in a direction to tile it in one dimension, mod-C select, click hold walk to extend this over the other direction.
This just sounds exhausting.
I'd rather try to sneak into a hyper-religious xenophobic town and get captured and turned into servant labor at a mining camp. And I'm not even taking about visiting Texas!
It's basically just Ctrl+C and Ctrl+V in terms of complexity and ease of use. Factorio is very ergonomic and the only way I can interpret the abovementioned "tedium and repetition" is that they never played long enough to get to the bots stage.
Factorio rewards the players progression through the game by taking away points of pain. The helper bots being a key transition. Without that pain there would be no reward. The fact that refactoring is expensive in time and complexity is what makes it realistic - it makes the earlier decisions important and consequential.
You can always enter creative mode and make whatever changes you want in the game instantly, but in my experience that does make it less fun.
It's basically programming with really bad ergonomics. I'm a programmer by profession and I was immediately frustrated by Factorio because I realized I'd rather just write code and build a project because it would scratch the exact same itch while actually being useful.
I tried really hard to like Factorio, and I bounce off of it each time after playing a save for 4-8 hours for exactly this reason. I know a lot of programmers who love it, so I almost wish I could get into it.
Now I'm tempted to try again, because it feels like the kind of game I should really enjoy.
The biters are just a line item in the spreadsheet. Early players might build a wall with an ammo belt, for example. There's lots of options that don't involve responding in real-time.
Though these days I just manually clear the pollution cloud with a pocket full of turrets. The forward deployed engineer model, if you will.
The nests stop being a concern once you have significant production and a few hundred logistics bots. Then you just build walls beyond the pollution range and place a gun turret every other square (or lasers at level 10 or above). I only use nauvis for uranium now because my Aquino production is weak (fusion). Late game you rarely go back to nauvis, just send your fleet of platforms to resupply reactor fuel autonomously.
The issue is most of these play-throughs happen on “normal” response speed. Way too slow for these LLMs to send back and forth and process this many tokens in a quick enough reaction time.
I’ve yet to see running these LLMs on Ultra-fast (8x speed on Sol). Wondering if that’s fast enough to do real time
I've been giving my little games to text to both Qwen 27B and Qwen-Next, and they both try to debug bugs they encounter unprompted. So I have to specifically mention to them to not try and ghidra my game and just play it as intended or report a bug in BUGS.md file and carry on.
I've been watching streams of 9 different models playing Factorio together [1] and it seems the answer is — not very well at all. They run into the same issue they have elsewhere in the game as well as in coding tasks: inability to create structured plans for the future.
So they get into a cycle of: 1. Get attacked 2. Build a few turrets 3. Get attacked. They don't consider buffing up their defenses or going out to deal with the threats or preemptively placing turrets when they expand. Not without human probing.
Also, they had the same issue with hand carrying resources around as some other comments here mentioned Sol did.
It is weird and almost alien how AI designs factories. When Claude announced Fable 5 they also did a Factorio demo that was the same kind of designs and a lot of hand feeding and direct insertion.
In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution. But I honestly expected more from frontier models
To them the player character is just as good as a belt for transporting materials.
The player character actually does have an extremely high capacity so if you are constantly running between chests your factory will still be reasonably high throughput.
What are you talking about? Maybe at the beginning, yes. Everyone carries stuff around when you have 2 assemblers, but once you start building your bus, the throughput of the player is orders of magnitude worse than using belts (or, if you're into that, bots)
> In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution.
Is the weirdness of the designs meaningfully beneficial in any way that the AI has figured out, or could it for example be that the training data the LLMs learned from are too different from the type of data it would need to learn from in order to interface properly and play the game better? And then because of that incompleteness, the LLM has to hallucinate a lot and it results in weird outcomes.
I think one of the problems is that AI has no sense of laziness or time. Unless it's in their training data or prompted they're not going to bother automating something
Actually if you estimate the time required it's plausible that player feeding comes out on top, especially if setting up automation is not that obvious. But human player would never play that way, they would get bored from the repetitiveness, and the micro scheduling would be a headache. So the game designers never really have to close that gameplay loophole (or closing it would make human experience worse).
I have a similar problem on my GK2 vibegamed playthrough (see https://news.ycombinator.com/item?id=50302771), if I want computer to play similar to how a human would, I need to artificially close that loophole (for example, AI prefers to gather/eat low-tier food to cooking better).
Anyone making this claim just hasn't played a lot of factorio, because this just isn't true. Player feeding is fast in the early game, but it just doesn't scale. There's too many buildings to feed and it takes too long to walk between them. People have already made tool assisted speedruns, and as expected it involves building a huge automated factory as quickly as possible.
I said it's plausible. Perhaps not for basic materials, but things like robot frames, higher tier research packs, modules, anything that has low component amount compared to time or amount required for crafting.
Even speedrunners do quite a bit of handfeed. And if AI/player cannot figure out the automation fast enough - the handfeed might easily come on top.
AI playing video games is a funny twist of faith. We call the AI-controlled characters NPCs, but now we've become NPCs watching AI play itself.
How weird is that?
At the same time this might help AI learn better - each game is a new system, a new set of rules to make sense of, each one has a unique tutorial aimed at human intellect to keep engagement, as well as teach you how to play.
As much as games are similar, especially within the same genre, the nuances are important.
I'd personally be impressed the moment the AI manage to play the games following those newances and being efficient as well as effective.
> The rocket launch was only achieved through significant ‘manual’ effort on Sol’s part as it papered over its late-game failures to connect certain production systems by repeatedly ferrying supplies between areas of the factory.
Looking through the video, around 12m one can see the Sol trace report the cancellation of a conflicting 'courier' (hand-feeding) script.
Rather than implement 'manual' effort, it seems like Sol considered the Engineer to be just another unit of automation!
That's not exactly within the intent of the challenge, but it's not too far out of bounds either. It's also a hard loophole to close if the agent interacts with the game through a scripting layer.
Nick Winter has been streaming a game of Factorio with the major labs' bots for, I wanna say weeks now. They made it off Nauvis and are now roaming space. It's pretty interesting to follow, though the commentary's been a bit quieter as of late.
Honestly, I'm more interested in the harness. Having an LLM gardener that could expand outposts to connect to the main network, or just pop in to observe if there's any major problem to report could be a cool project for long-running multiplayer bases.
I want to design high-SPM factories. I don't want to be interrupted each time a stone patch runs out.
Second point: multiplayer runs in real-time, but I have real-life obligations too - having a short report along the lines of "all quiet on the eastern front" or "actually, Gleba got stuck and Stompers took over" helps.
There's still plenty of game to be played with some of these more complex QoL features.
I don't have the budget to do so, but if I do, it would be interesting to put different models in the same game with the same prompt to observe the behaviors that different models have when told to problem solve.
I bet it helps that Factorio attracts exactly the kind of obsessive players that'll leave a digital trail of training data documented in precise detail
Bit like warthunder guys can't help but leak classified info
This is a much better benchmark as a concept (without knowing the details of how the problem is framed to the agent) than some of these ridiculous things the frontier labs are reporting (vending machine simulator? come on).
Can you report more in depth at stumbling blocks? It's a surprisingly linear curve.
I'm going to try it on a local run.
I wonder if there is room for "recommendorio" where it gives you AI advice on your current factory.
Is the model pre-trained with any knowledge of the game?
> The rocket launch was only achieved through significant ‘manual’ effort on Sol’s part as it papered over its late-game failures to connect certain production systems by repeatedly ferrying supplies between areas of the factory.
Robots: they're just like us!
> Robots: they're just like us!
Hah I dunno, I think most human players wouldn't find that very fun and would instead take that as an opportunity to refactor the factory for optimized performance (i.e. the core gameplay loop)
Ish.
When you just need a few parts to be made, it is sometimes very tempting to drop a chest, fill it up with parts, then go do something else, one of the 1000 tasks you have been neglecting.
That's definitely true, but only to a certain point, after a while it becomes too monotonous to dig around in chests building and combining stuff in an ad hoc manner. LLMs have no problem with that kind of repetitive minutia though.
By end game you can just have blueprints and robots build everything for you.
Logistic bots, once they are unlocked, the game becomes trivial.
Once you make a few blueprints, then the real joy of factorio is found. Nothing like splatting down an outpost and watching it sprint into existence as thousands of “agents” build it out.
my husband plays exactly like that, and whines until I log onto the multiplayer server so that I can "make it work like a programmer".
I then log in, create a bunch of logistics chains, and then a few days later he whines about it again when he enters the next phase of gameplay.
This continues until fast logistics drones, then he takes over; less need to consider routing & order .
Factorio is interesting, a lot of players just treat it like they're playing Postal and just want to gun bugs down in creative mode equivalents, meanwhile i'm fretting about green boards per second.
New players can find refactoring the factory to be more difficult.
Yes and no - for a one-off I will just craft something "by hand" instead of doing a ground-up rebuild. (The exception, of course, was my build for the "Lazy Bastard" achievement.) I don't go into it thinking "oh boy it'll be fun to mule around these items across the width of my factory!" but it's also not so offensive that I strictly automate everything all the time. Not that I play the game entirely the way it's meant to be played
Naturally, "I'll do it once" starts as a temporary crutch and ends up a load-bearing liability. A phenomenon entirely absent from our day jobs ;)
i think its more of an effect of factorio's hidden information.
youre not really sure how to set up some of the logistics/how much of what you will need where ahead of time
What information is hidden in Factorio? I can't think of any stat that isn't shown to the player in some way.
He is not talking about stats.
When you start a new game in Factorio, if you never played it, you can’t know how much of each resource you’ll have to extract and neither how much of an item you’ll need to produce.
I know it’s easy to forgot after thousands hours of playtime, but the game starts with your character having to manually mine coal one by one.
> When you start a new game in Factorio, if you never played it, you can’t know how much of each resource you’ll have to extract and neither how much of an item you’ll need to produce.
Some of this information should be recoverable somewhere in the tech tree tooltips, if nowhere else.
More importantly, vanilla Factorio plays a nasty little trick on first-time players: The final sciences and launching the rocket both require scaling up. This means that either you refactor your factory, or you grind out that first launch super slowly using an inadequate production line.
Veteran players know ways to design some modularity into a factory, and when to yank out subsystems and rebuild them bigger somewhere else.
Factorio: Space Age is actually more forgiving about scalability. You don't need to scale up as much, and you have much more powerful tools for doing it. Instead, Space Age has lots of novel game mechanics to master, and higher total system complexity across the solar system.
Not true, every item shows its raw materials right there in its menu. The rest is just arithmetic.
Also the LLM probably has seen dozens of factorio-related websites and setups.
And also, player (artificial or natural intelligence) don't need to setup everything upfront, it's actually the main gameplay feature is to run between production areas to expand/optimize them.
Technically every item displays requirements and crafting time, but most human players will just eyeball production rates until they start trying to build larger scale factories that you can't easily jerry-rig fixes onto.
Possibly there is a lack of item-throughput figures for belts and stuff though.
Thats when you build a giant bus and select what you need out of it.
A bus doesn't solve this, because it still is contained by items/second. You can't just select copper, if there's no more copper on the belt. And you might not be able to jerry-rig another belt of copper onto your bus.
A bus doesn't solve the issue the person you replied to was talking about.
The answer is all of it and yes.
I still don't get the point. The information is there right from the beginning. For humans itS just tedious to calculate it and why should one?
Then also, you can finish the game even when not planning ahead.
That information is certainly in the training data.
IME the models know some basic information about all the popular games, including some basic strategy, but they don't know/remember the details to be even a moderately familiar player. And usually if you ask a question they will google it.
Yet, that's not how new players (human or AI) approach the game. If you look at AlphaZero, probably state of the art in learning e.g. chess (more information about rules than Factorio has in game) from scratch, using only rules, it just plays a lot of games with itself. It doesn't "just think a lot" to come up with a perfect strategy; it experiments.
The AI has trained on the entire factorio wiki along with every tutorial video ever made and who knows what else. I don’t see room for hidden anything in that scenario.
Models don't have perfect recall
If LLMs worked that way, you would never have to search for prior art again.
How do you mean? (I have played 1 hour of Factorio)
I am not the OP but I have (many) hours in Factorio and I think what they might be talking about is things like how it’s not apparent at the start just how much copper infrastructure you will need by the time you get to rockets and space platforms because the early recipes are iron-heavy. Or how important green circuit scaling will become. Or how much oil cracking might change your layout.
If you know what’s coming you make different choices in how you build that heads off a fair bit of refactoring.
I didnt think was possible to play for less than 1k hours.
Those are rookie numbers, you need to pump those numbers up…
We found the business guy.
What do you mean? Factorio is an extremely calculable game and tends to show most of its information (ratios aside, those are in the wikis)
Unless I misunderstand, ratios are second-order and can be calculated; production rates are presented in-game.
If one provides it access to the wiki and reddit, if just from within game it's quite cryptic
It certainly sounds like an actual factory in my experience
If you haven't been keeping an eye on LLMs playing games, GPT 6.1 Sol can new beat games 100% vision-only (without harnesses that give the agent extra information about the game). Pokemon is no longer a difficult challenge for it, even with additional difficulty restrictions such as hard mode and level caps. Recently, GPT 6.1 Sol beat Final Fantasy VII and its bonus bosses vision-only surprisingly quickly.
I suspect Opus 5.5 can do even better but there aren't as many people testing it.
Where are the recordings of these games? I know this kind of stuff must be happening, but it seems oddly difficult to track down the actual recordings, which feel like the really fascinating thing people should be publishing if they’re putting in the time and tokens to do stuff like this.
Like I want to see what Pokemon strategies it uses: Does it try to brute force with its starter? Does it try to load all of its HMs onto one or two Pokemon? Does it use Flash? If it needs to catch a new Pokemon to beat a gym leader, how much does it remember about which species it encountered where and what their levels were?
The LLMs no longer brute force a starter, and they have to play competently, including the use of setup. Here's a screenshot of the completion of the GPT-6 Astra run in 21 hours with comically underleveled Pokemon (and it actually savescummed RNG!): https://www.reddit.com/r/ClaudePlaysPokemon/comments/1w79vim...
Another run with Ironlocke rulesets for Crystal (hard mode, level caps, permadeath) has even more diverse teams: https://www.reddit.com/r/ClaudePlaysPokemon/comments/1wf6zx0...
It has been a few years now. By now you should have realized that AI is for doing stuff at a massive x100 productivity boost and never getting a complete project done. (For some reason there's those "final kinks to iron out", once the AI figures out where it fucked up. Just two more prompt modifications.)
If you reading every wiki, forum, dev blog/patch note, watching every video, etc before playing, I’d be more surprised if you could NOT play a game.
Exactly. All of these examples are well documented games. And even then it’s only really good at games that don’t need realtime.
You could make the same argument for these LLMs years ago but there's a reason that they've become faster and more accurate at beating these games. GPT 5.5 took 218 hours to beat Pokemon Fire Red; Astra took 21.
Only months ago did these LLMs make it past the Victory Road boulder puzzle without harness assistance.
GPT-4 was trained on all that too and couldn't get any task at all done
I would be willing to bet it can beat games un-released at training time as long as latency is not a requirement, which does limit the game types a bit.
How do they feed the game screen to the models? I imagine the models are too slow if you send 60FPS, but I feel like that’s also the case if you maybe feed it 1FPS? If you go any slower, you’ll miss too much info?
I would assume it's pull-based so the agent takes a screenshot whenever it finishes a turn.
Essentially yes. For games with some real-time component (FFVII), the game pauses after actions are executed to take a screenshot and let the agent think.
Your last sentence mirrors my model choices recently. Opus can do things very well but Sol can do them well for less than 1/4 the price.
And 10x the time.
Sadly this.
Sol, write me a function that moves all the toothpicks from jar 1 to jar 2
>computing requirements
>planning
>writing test harnesses
>writing tests
>450/450 offline tests pass
>1 toothpick moved...
> verifying
>writing acceptance tests for toothpick 1
>error in acceptance test, fixing...
>toothpick is one pixel from expected destination, restarting toothpick job
>...
>THIS MODEL IS AT CAPACITY. PLEASE TRY ANOTHER MODEL.
It's not a mystery, artificial intelligence ranks opus exactly as 1/8 to 1/4th cheaper.
Opus and claude models have always been lagards in vision. It just hasn't been a focus like coding for anygropic. Gemini and ChatGPT have always been much stronger in this area
What's "anygropic"?
Close on the keyboard to and small Levenshtein distance from Anthropic
typo for Anthropic
I would like to see it play Slay the Spire (or 2) and beet it consistently. Mainly to see what strats it uses. I suspect it wouldn't have too much trouble.
Some of these Games are definitely in the RL environments.
Do you have link for the VII bonus boss run?
The channel is here, with past broadcasts: https://www.twitch.tv/gpt_plays_games
Watching AI play games is fun, but I can't wait for someone to crack having small LLMs in games to enrich them in useful and interesting ways.
Not sure about LLMs but I yearn for someone to train neural nets to play the NPCs in a shooter game well but without having aimbots or seeing through walls - basically a “true” version of F.E.A.R. AI where they play simulated squads that overwhelm the player with various tactics and adapt to their playstyle. Or even just make increasingly complex state machines and goal oriented systems which might be more performant but still encode the main rules.
You want to create a human feeling using AI, but AI is not needed for the feelings you want to create. All you really want is a feeling of variety and surprise in the way computer opponents behave. A LLM-level of AI might just do the same thing all the time and become quite boring.
And yet, that argument I just made often bothers me, because I do think there is a place for strong AI in games, and many people just reflexively say "strong AI wouldn't be fun". I think they are partly right, but I also think a strong AI that is purposely dumbed down in fun ways could be really enjoyable.
When it comes to shooters though, those games aren't really about competition and decisions, they are about giving the human player an experience. I do think strong AI is overkill in such situations.
A strong AI for chess absolutely improves chess computer games.
A strong AI for a Magic: The Gathering style card battler would improve the game.
A strong AI for Civilization would improve the game, but I think we're starting to get into a realm where you can fake some things. I do think Civilization games are lacking because of poor AI, but I don't know that we need LLM-levels of AI in Civilization.
Again, in shooters, there's no decision, no illusion of fair play or equal competition based on making decisions, it's all about the human experience, and so LLM-level AIs aren't needed here at all.
> You want to create a human feeling using AI, but AI is not needed for the feelings you want to create.
> When it comes to shooters though, those games aren't really about competition and decisions, they are about giving the human player an experience. I do think strong AI is overkill in such situations.
> Again, in shooters, there's no decision, no illusion of fair play or equal competition based on making decisions, it's all about the human experience, and so LLM-level AIs aren't needed here at all.
No, playing against humans absolutely sucks to an untold level for me - because more often than not at some point it ends up being about who has better aim or reaction times and I lose like most of those.
Stuff like that makes everything from CS, Valorant, Battlefield, Wardogs, Active Matter, Tarkov, ABI, Insurgency Sandstorm and any other shooter game unplayable. Meanwhile purely skulking around corners is no fun and also sucks, because that still doesn't resolve the imbalance, and that's after over a thousand hours and sometimes with SBMM and all. Teamwork can't really fix that, even when I tried playing a healer in Valorant friendlies would get taken out, I'd get picked off, round over, we lose.
What I really want is AI that is worse at aiming than I am but is more numerous and makes up for it with numbers and utilizing those numbers well with tactics. Games like FEAR and Trepang2 and Incursion Red River evoke that feeling, but could definitely be made even better in that regard! It would be strong not because of sniping me from across the map like a lazily made bot, but rather because there'd be grenades going out in my direction, they'd use cover well and suppress me while some others push from the flank, not just rush me with godlike aim.
That is before you consider playing online being a minefield (especially in Eastern European servers), it's a coin flip whether someone in Wardogs will be trying to flip choppers or AFK bot at base, or whether they'll decide that being annoying and sharing the best slurs they know is what they want to do today. It seems like there's a narrow band between tryhards who berate you for not playing good enough and people who are just trolling. I don't want that kind of hit or miss experience, ever. It's even the same with Arma RP servers (like Altis Life, the idea is cool) when I tried those - you'd be able to make money by mining and processing ores or whatever for a bit, until someone would randomly shoot you for no real reason.
Perseverance might be the drive missing from those bots.
I expect the area of fun between painfully easy to beat and infuriatingnly overpowered with superhuman skill level to be quite narrow.
I have been playing with this. Having tiny models provide more contextual information when an NPC speaks is cool but the "small enough" models tend to go off the rails.
It's fun if you have a handful of people playing, but it would quickly go towards negative reviews if you tried to sell it.
I don't think an LLM should ever be having a two-way conversation with the player. It would be more suited to 2 npcs having conversations amongst themselves. All dialogue delivered to the player should be human written.
At most it should be mildly tweaking existing dialogue, e.g updating a line to reflect game events. But as soon as you present a prompt window to the player the entire illusion that you are in a game world instantly dies.
> I don't think an LLM should ever be having a two-way conversation with the player. It would be more suited to 2 npcs having conversations amongst themselves. All dialogue delivered to the player should be human written.
I don't agree. All -important- dialogue yes. Background NPC chatter? Even when studios spend a lot of effort crafting multiple lines, pretty soon you have seen them all. They also don't change based on player actions.
i think it can work for procedurally generated situations, and the llm is acting more like a translator between data model and player rather than being a data storage layer.
thats close to tweaking existing dialog, but with a bit more freedom.
you still dont have to give the player arbitrary response
You absolutely cannot let the player get any whiff of AI slop so the integration needs to be extremely delicate. Watching a film and noticing a person has 6 fingers in a scene would instantly kill any engagement someone would have with it. For it to remain as art it must always be fundamentally about human-human communication, I want to have a conversation with the game developers through the game, not with a LLM. If developer intent ever becomes muddled the game has failed.
Most dialogue in games is crap. I would much rather have an LLM writing the dialogue so that it can include references to player choices over what currently exists.
Nothing great was ever built by someone who was sweating the reviews. Fuck 'em, do what feels right to you.
I built a factorio "sidecar" LLM harness that would look at production stats every few minutes and identify issues. Even sonnet was good enough at finding potential things to alert on. coal production dropped and then iron production dropped two minutes later --> top potential issue.
Not useful enough that I actually play with it, but it was capable enough to be useful even just with sonnet.
It's a funny situation at the moment because most attempts are just sort of shoehorning LLM chat into a game that doesn't really need it, and then gamers quite rightly mock the results which don't really add anything to the game.
I think there's a bit of a lack of knowledge of game history on both sides, because we already know it can be fun when a game is designed around it. It worked in Façade (2005!) and it worked in Starship Titanic (1998!!), and both of those could have worked even better with the technology we have today.
One type of game that seems like an obvious one to me, is something a bit like The Sims except that the characters' decisions and speech are LLM-controlled. Maybe one of the characters is player-controlled, so you can interact and talk with everyone else. Or to describe it from the other direction, it'd be a kind of interactive fiction like AI Dungeon invented, but grounded in a physically simulated setting.
I sometimes wonder if Façade actually used neural networks for text generation or just simple markov chains... I remember pewdiepie saying melon and getting kicked out lol. Also the Sims would be awesome but imagine an RPG with rich worldbulding, and LLM's interacting with the lore, like in an Elder Scrolls or Fallout title. Wouldn't it be mind blowing?
There has to be a AAAA studio building the new, greatest MMO, that also utilizes LLMs as part of the game loop, surely?
(Not sure how prohibitive it would be to do anything interesting with local LLMs of any kind in a big game, given how many platforms/setups you have to make the stuff run smoothly on, and the sheer amount of RAM they would bind)
https://proxiai.com/ Will Wright (The Sims, etc) is dabbling with AI in his latest game. No idea if it'll be any good, but there's def companies out there trying.
AAAA studios don't take risks. Financing a top game is way too expensive to gamble on. Only indies and small studios working on cheaper games can risk trying unproven ideas.
not taking risks is the biggest risk in the times when the most anticipated game drop (gta 6) is probably going to be decompiled within days if not hours of release
In "Where Winds Meet" you can talk to NPCs
With the aid of SOTA LLMs, you can encode specialist intelligence into computer programs without going through the roundabout route of training small LLMs. And even when you do go the route of ML, it doesn't necessarily have to be an LLM or even a Neural network.
I hope to experiment with using classic tabular reinforcement learning in game AI one day. Imagine a framework that makes setting up small Q-learning tables easy, and you can just sprinkle dozens of Q-learning algorithms throughout your traditional AI scripting.
These old basic algorithms often have mathematical proofs that they will converge to optimal behavior, and they do so fairly quickly. Turning the complex world of the game into a Q-table would require some creativity and judgement, but is not that hard I think.
In the end we would have a controllable AI that actually does adapt. That's my hope anyway.
I cannot wait for NPCs having a casual conversation. Doesn't seem too hard, as current chatbots are already role-playing.
I understand https://neal.fun/infinite-craft/ uses an LLM under the hood each time a novel combination happens.
This... Hopefully inference can become so cheap games can have simple LLM's integrated into them. Imagine an RPG with LLM's and with rich worldbuilding; like Fallout NV with LLM's. Honestly, scary!
Inference is already cheap enough. It's basically free for the quantities people actually want to read.
LLMs only become expensive if you let them generate 1000 lines of code every 2 minutes.
Fable seems to be a potential franchise to do all this; imagine a "living fantasy world" where everything truly unfolds based on your decisions, giving NPCs all the functions/actions that a player has.
i think hooking in an llm to dwarf fortress for turning the very mechanistic descriptions into prose. same with the adventure mode talk. instead of a constant "well weve got beasts" an llm could read a bunch of the language tree to make a unique introduction for each.
same with the df artworks. get kitfox to design some art styles for dwarves, elves, goblins, and humans, and then instead of just reading
"this is a fine engraving of a dwarf. it concerns the events of 102 when ulrist fastbuttons was elected mayor. ulrist fastbuttons is sobbing"
SOTA LLMs would probably be fun opponents in turn based 4x.
Will be interesting to find out what they can bring over simply having better SOTA LLM coded opponents (I am sure, we will).
LLMs could be interesting geopolitical roleplayers if instructed to faithfully assume the leadership traits or cultural biases of the their intended role, especially in games with expressive diplomacy systems like Stellaris or Europa Universalis. It could also make the diplomacy experience a lot richer because the LLM is able to interpret ligusitic tone and other verbal cues. The modern diplomacy experience is basically just a dialog tree with stats and dice rolls attached to it.
Unfortunately for it to be sustainable it would need to be an LLM with VRAM usage probably under a couple gigs max so it can run locally on the gaming PC. That said it may well be better than current AI in strategy games.
Yes, the insane hardware requirements make it totally impractical - that's why I say "turn based" because turn based games would at least allow for a practical gameplay experience even with remote LLM APIs.
I bet a small 1B model trained solely on how to play the strategy game would do better than current AI solutions for these games and that would probably be able to run even on integrated GPUs.
I would love the revival of the RTS genre with better computer enemies.
Such AIs have existed in Brood War and Dota 2 for a very long time now (pre-LLM), problem is they aren't very fun for humans to play against.
bah, such a pity we didn't have those in wc3 times. bots sucked badly.
I could play online, but my dial-up costed $1 per hour for 250ms latency and near guarantee of at least one disconnect per hour.
with better being more convincing, rather than just stronger
its the same for like a chess bot. i dont want a perfect stockfish that occasionally makes a serious blunder when a dice rolls a certain way. i want one that actually falls into traps
You could play against an AI right now on the Brood War ladder right now. Granted, you have to make it to S rank first because that's where the bots end up.
SS13 but the ship's AI is actually an AI instead of a player in disguise
impossible, unfortunately, because LLMs suck at creative writing. without handholding, they are incapable of producing anything but cliche-ridden purple prose for young adults. you can feed it the first half of a book with lots of sex, violence, and profanity, ask it to continue the story, and it will degrade into safe slop within two chapters.
both American and Chinese labs aim their models at coding and knowledge work, so they penalize adult themes in post-training.
There are already AI-based mods for games like Grand Theft Auto V that make NPCs talk back and follow your verbal commands: https://www.youtube.com/watch?v=3_Eng-F4akI&t=106s
A little unpolished because it was bolted on by fans and not the original game developers, but otherwise very impressive.
One limitation of these type of demos (and similar with Minecraft, etc) is that: 1. Putting into "peaceful" mode makes response time a non-factor 2. These models have read and have access to every guide and video ever published on the subject. Most humans, especially on first play, learn game systems and interactions from scratch, then develop novel strategies and plans and "hypothesis test" their way into beating the game.
This is still impressive compared to where we were a few years ago, and being able to synthesize all existing information about a game and then play it to completion in a semi-realistic setting is great.
At a quick glance someone might mistake this for "brilliant AI solves Factorio", when in reality, given a similar but different game where it wasn't gifted all the information or training data (e.g. massive videos and screenshots) it might fare terribly (as others have pointed out in comments).
>These models have read and have access to every guide and video ever published on the subject.
This has been true for every model since 2023. It's not nearly as big of a factor as people believe it is
Isn’t the point of arc agi 3 to measure general game intelligence even if it’s not a AAA game? Frontier models have obliterated that. And they do keep most of it private so no gaming there.
The models these days have 3 components that is for sure going to get better and better
1. Very good vision (it can identify almost anything in a picture) 2. Keyboard and mouse actions (slow but it does it well) 3. General world understanding (this one is toss up but given Astra or Fable, it surely does well even if it takes a while)
Now combine all these with specific RL environments that may use specific games which also generalizes to an extent and we have a system that can only get significantly better from here.
As for speed, I’d say it’s a question of how much money you are willing to throw (like Cerebras for example).
Good point. There are also a huge amount of open source calculators, planners and other tools that these models have undoubtably been trained on [0]
There are full blueprints for factories online available to copy-paste in its training data [1]
I wouldn't be surprised if you can beat the game by just copying a bunch of blueprints from the web and chaining them together.
This is an old game relatively speaking and its been optimized to hell.
0: https://factoriolab.github.io/2.1
1: https://factorioprints.com
These are really two different challenges - playing the game for the first time with exploration, and beating the game with all the prior knowledge.
Shhh you can't say anything anti-hype here.
It doesn't seem to understand the core mechanic of Factorio, which is putting outputs onto belts and routing them to other inputs. It does this in a few places, but for the most part everything is put into chests and moved around by the player.
What if the player's inventory has a higher throughput than belts?
Part of the reason players try to automate manual feeding is the annoyance factor, which AI doesn't care about.
Handfeeding only has higher throughput for single trips/a single item type, definitely not for an entire factory. The player inventory and movement speed can't keep hundreds of assemblers working continuously. A high throughput starter factory looks something like this:
https://youtu.be/RI4ocNajf_w?si=O1aRi22riKwiuh_p&t=7493
Back of the envelope math:
If you measure by the product of throughput and distance traveled, for an item with a stack size of 100 (e.g. metal plates), the player's inventory adds up to roughly the same capacity as 2500 yellow belts. And that's assuming 100% utilization running back and forth in both directions, with no unused inventory space and no wasted time.
A basic assembler can manufacture 2500 belts in about 21 minutes. So as a rule of thumb, if you're playing for longer than 21 minutes, belts are worth the investment (at least for bulk transport of raw materials).
(I guess maybe add an additional factor of at most √2, since the player can run diagonally and belts are orthogonal.)
A T1 belt throughput is 15 items/s. Player inventory is 80 slots (2 cargo wagons, incidentally, though players load/unload instantly).
Stack size varies, but let's take 75 as an average, which means a player can carry 5600 items.
But carrying isn't throughput. If a round-trip takes 30s, player throughput will be 186 items/s. That's a measly 12 T1 belts.
Well, the 1 hour TAS speedrun is probably close to optimal and it also does lot of stuff that would be annoying to do in real human gameplay: https://www.youtube.com/watch?v=tImNejO5g48
If you don't have enemies, you have unlimited time. Doing it thru player is what it learned and wasn't pressured to do better (it has no fatigue as opposed to humans).
Coding agents are bad at architecture, Exhibit no. 12057.
It's just easier to do manual stuff when the complexity of scaling the automation becomes cognitively demanding. A new player does mostly everything manually while experienced players have a natural intuition for planning and scaling the automation.
To be fair, I do do the same.
Factorio is a much easier game without the bugs being a pain in the arse as you expand - i wonder how Sol would handle the bugs
The biggest thing the bugs introduce from an LLMs perspective is a necessity for realtime response. Without them, the LLM can take as long as it wants to decide and do its next operation on the factory. With them, not responding fast enough results in death (particularly early game where a lot of biter combat is player driven). I presume it doesn't do great at this.
Glad to hear we've got LLMs controlling military drones in real life :(
Someone got chatgpt to drive a car through a mock driving course.
It took about 5 minutes for it to drive 50ft.
We're not quite there yet
I don't think we do.
you could pause the game between actions, while the model reasons about what to do next.
That was exactly my thought when reading the headline, before even looking into the article. Sure it's still somewhat complex to get to a rocket launch, but without the biters where is the challenge, other than complexity? It's like a big excel sheet with conveyor belts, pipes and electricity poles at that point only.
The more interesting challenge is the one you eliminated, complexity, in getting efficient logistics and arrangements. Dyson Sphere Program (until late in its existence) and Satisfactory don't have waves of enemies attacking your bases but remain challenging and a satisfying puzzle. I remember I used a constraint solver to help me optimize some late game recipes for Satisfactory.
> where is the challenge, other than complexity
Patience, for one. I reach a point where I'm looking at a mess of coal-fed sorters that need a complete rip-and-replace with electric ones, and that's usually the point where I close the game and go back to something easier like Kenshi.
I know it's a pipe dream, but these kinds of games (Factorio, Satisfactory, etc) would all benefit from sort of brain-game interface where I could just think about what I wanted to do.
The tedium and repetition is really what kills the enjoyment for me. The problem solving, planning, forethought, etc are super fun but it's killed by the former.
Blueprints and construction bots, plus the well-thought-out and extensively battle-tested set of keyboard commands for them, are basically a brain-game interface for Factorio on the level that Vim is for text. For the grandparent’s “rip and replace coal burners” I am hitting mod-D and dragging a selection box over the offending area, then hover-Q on a nearby element I want to use and shift-click to plan it, once I have planned one instance of a subpattern its mod-C select, click hold and walk in a direction to tile it in one dimension, mod-C select, click hold walk to extend this over the other direction. It is now done, the bots will handle everything else - because Wube are an incredible bunch of UX engineers, I can place the new building before the old buildings are even removed, leave, and trust that everything will be done correctly. Highly recommend getting a “Starter Bots” mod that feels appropriately balanced.
> I am hitting mod-D and dragging a selection box over the offending area, then hover-Q on a nearby element I want to use and shift-click to plan it, once I have planned one instance of a subpattern its mod-C select, click hold and walk in a direction to tile it in one dimension, mod-C select, click hold walk to extend this over the other direction.
This just sounds exhausting.
I'd rather try to sneak into a hyper-religious xenophobic town and get captured and turned into servant labor at a mining camp. And I'm not even taking about visiting Texas!
Much like Vim, it is exhausting to read a description. But it’s like a second of muscle memory that happens instinctively.
It's basically just Ctrl+C and Ctrl+V in terms of complexity and ease of use. Factorio is very ergonomic and the only way I can interpret the abovementioned "tedium and repetition" is that they never played long enough to get to the bots stage.
Factorio rewards the players progression through the game by taking away points of pain. The helper bots being a key transition. Without that pain there would be no reward. The fact that refactoring is expensive in time and complexity is what makes it realistic - it makes the earlier decisions important and consequential.
You can always enter creative mode and make whatever changes you want in the game instantly, but in my experience that does make it less fun.
It's basically programming with really bad ergonomics. I'm a programmer by profession and I was immediately frustrated by Factorio because I realized I'd rather just write code and build a project because it would scratch the exact same itch while actually being useful.
I tried really hard to like Factorio, and I bounce off of it each time after playing a save for 4-8 hours for exactly this reason. I know a lot of programmers who love it, so I almost wish I could get into it.
Now I'm tempted to try again, because it feels like the kind of game I should really enjoy.
The biters are just a line item in the spreadsheet. Early players might build a wall with an ammo belt, for example. There's lots of options that don't involve responding in real-time.
Though these days I just manually clear the pollution cloud with a pocket full of turrets. The forward deployed engineer model, if you will.
Its puzzle solving. For a lot of people, that _is_ the fun part.
The nests stop being a concern once you have significant production and a few hundred logistics bots. Then you just build walls beyond the pollution range and place a gun turret every other square (or lasers at level 10 or above). I only use nauvis for uranium now because my Aquino production is weak (fusion). Late game you rarely go back to nauvis, just send your fleet of platforms to resupply reactor fuel autonomously.
The issue is most of these play-throughs happen on “normal” response speed. Way too slow for these LLMs to send back and forth and process this many tokens in a quick enough reaction time.
I’ve yet to see running these LLMs on Ultra-fast (8x speed on Sol). Wondering if that’s fast enough to do real time
"AI found decompiling game to fix a bug after being instrcuted to beat it" sounds like an IPO friendly headline.
I've been giving my little games to text to both Qwen 27B and Qwen-Next, and they both try to debug bugs they encounter unprompted. So I have to specifically mention to them to not try and ghidra my game and just play it as intended or report a bug in BUGS.md file and carry on.
I've been watching streams of 9 different models playing Factorio together [1] and it seems the answer is — not very well at all. They run into the same issue they have elsewhere in the game as well as in coding tasks: inability to create structured plans for the future.
So they get into a cycle of: 1. Get attacked 2. Build a few turrets 3. Get attacked. They don't consider buffing up their defenses or going out to deal with the threats or preemptively placing turrets when they expand. Not without human probing.
Also, they had the same issue with hand carrying resources around as some other comments here mentioned Sol did.
[1]: https://factorio.nickwinter.net/
It is weird and almost alien how AI designs factories. When Claude announced Fable 5 they also did a Factorio demo that was the same kind of designs and a lot of hand feeding and direct insertion. In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution. But I honestly expected more from frontier models
To them the player character is just as good as a belt for transporting materials.
The player character actually does have an extremely high capacity so if you are constantly running between chests your factory will still be reasonably high throughput.
What are you talking about? Maybe at the beginning, yes. Everyone carries stuff around when you have 2 assemblers, but once you start building your bus, the throughput of the player is orders of magnitude worse than using belts (or, if you're into that, bots)
For the purposes of launching a rocket the player character's inventory is vast.
> In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution.
Is the weirdness of the designs meaningfully beneficial in any way that the AI has figured out, or could it for example be that the training data the LLMs learned from are too different from the type of data it would need to learn from in order to interface properly and play the game better? And then because of that incompleteness, the LLM has to hallucinate a lot and it results in weird outcomes.
No their weird designs are not better in any way - yet.
I think one of the problems is that AI has no sense of laziness or time. Unless it's in their training data or prompted they're not going to bother automating something
Actually if you estimate the time required it's plausible that player feeding comes out on top, especially if setting up automation is not that obvious. But human player would never play that way, they would get bored from the repetitiveness, and the micro scheduling would be a headache. So the game designers never really have to close that gameplay loophole (or closing it would make human experience worse).
I have a similar problem on my GK2 vibegamed playthrough (see https://news.ycombinator.com/item?id=50302771), if I want computer to play similar to how a human would, I need to artificially close that loophole (for example, AI prefers to gather/eat low-tier food to cooking better).
Anyone making this claim just hasn't played a lot of factorio, because this just isn't true. Player feeding is fast in the early game, but it just doesn't scale. There's too many buildings to feed and it takes too long to walk between them. People have already made tool assisted speedruns, and as expected it involves building a huge automated factory as quickly as possible.
I played Factorio quite a bit..
I said it's plausible. Perhaps not for basic materials, but things like robot frames, higher tier research packs, modules, anything that has low component amount compared to time or amount required for crafting.
Even speedrunners do quite a bit of handfeed. And if AI/player cannot figure out the automation fast enough - the handfeed might easily come on top.
This is where human-directed looping revision fits in on a lot of my projects.
AI playing video games is a funny twist of faith. We call the AI-controlled characters NPCs, but now we've become NPCs watching AI play itself. How weird is that? At the same time this might help AI learn better - each game is a new system, a new set of rules to make sense of, each one has a unique tutorial aimed at human intellect to keep engagement, as well as teach you how to play.
As much as games are similar, especially within the same genre, the nuances are important.
I'd personally be impressed the moment the AI manage to play the games following those newances and being efficient as well as effective.
> The rocket launch was only achieved through significant ‘manual’ effort on Sol’s part as it papered over its late-game failures to connect certain production systems by repeatedly ferrying supplies between areas of the factory.
Looking through the video, around 12m one can see the Sol trace report the cancellation of a conflicting 'courier' (hand-feeding) script.
Rather than implement 'manual' effort, it seems like Sol considered the Engineer to be just another unit of automation!
That's not exactly within the intent of the challenge, but it's not too far out of bounds either. It's also a hard loophole to close if the agent interacts with the game through a scripting layer.
I tried a few recently on Civ 6 Deity, and they all lose pretty quickly.
Sounds like an experiment I'd like to give a try--what integration layer did you use to get LLMs to control the civ?
Just running coding harnesses like Codex, Claude Code, and Cursor locally and having them do computer use.
To be fair, Deity on Civ games is brutal. I'd be interested in where the cutoff is.
Doesn't the AI (the computer players, I mean) "cheat" at that high of difficulty level?
That's moot- it's a comparison vs humans who can beat it.
Nick Winter has been streaming a game of Factorio with the major labs' bots for, I wanna say weeks now. They made it off Nauvis and are now roaming space. It's pretty interesting to follow, though the commentary's been a bit quieter as of late.
https://factorio.nickwinter.net/
Automating fun was not on the top of my priorities but sure.
Honestly, I'm more interested in the harness. Having an LLM gardener that could expand outposts to connect to the main network, or just pop in to observe if there's any major problem to report could be a cool project for long-running multiplayer bases.
I see it at https://github.com/ryanmadden/autorio and https://ryanmadden.net/claude-plays-factorio/, but I'd like to ask - any plans to support blueprints? Or push it further into Space Age or dare I say.. pyanodons?
No matter the answer: very nice!
> Having an LLM gardener that could expand outposts to connect to the main network, or just pop in to observe if there's any major problem to report
Why even play the game?
I want to design high-SPM factories. I don't want to be interrupted each time a stone patch runs out.
Second point: multiplayer runs in real-time, but I have real-life obligations too - having a short report along the lines of "all quiet on the eastern front" or "actually, Gleba got stuck and Stompers took over" helps.
There's still plenty of game to be played with some of these more complex QoL features.
There are mods that address a lot of these: https://mods.factorio.com/mod/InfiniteOres
You could also automate base status reports via circuits: https://www.youtube.com/watch?v=39eYYwwuqyw
Idk, fair enough I guess but it feels weird to skip the automation part of an automation game by having an LLM play it.
Yea, I also didn't like how the bot ignored belts and shuffled things through chests and inventory.
I agree though - skipping automation/scaling is a no go, that's why I play Factorio.
I don't have the budget to do so, but if I do, it would be interesting to put different models in the same game with the same prompt to observe the behaviors that different models have when told to problem solve.
https://factorio.nickwinter.net/
I bet it helps that Factorio attracts exactly the kind of obsessive players that'll leave a digital trail of training data documented in precise detail
Bit like warthunder guys can't help but leak classified info
Those biters make life nasty brutish and short for the robot. https://youtu.be/6ZdsmWfKNJU
This is a much better benchmark as a concept (without knowing the details of how the problem is framed to the agent) than some of these ridiculous things the frontier labs are reporting (vending machine simulator? come on).
Can you report more in depth at stumbling blocks? It's a surprisingly linear curve.
I'm going to try it on a local run.
I wonder if there is room for "recommendorio" where it gives you AI advice on your current factory.
Is the model pre-trained with any knowledge of the game?
it could be interesting to have a player vs player mode in Factorio, and see two AI playing against each other
how much did it cost I wonder
(Author) I did this on the 'Plus' plan over a series of 5h quota segments so $20 for me, but at API pricing the tokens would have been $52
Can you share initial prompt or something? Is it ChatGPT app or codex terminal? I’m assuming you did not intervene at all
is that spaghetti factory i am seeing? :O
Wow, that's the saddest headline I read in the week Patrick Stewart died.
Space Age or classic? Space age is much harder.
interesting artifact of over investment and subsidies of AI and datacenters
Who determines "over investment"
the market, and investors opinions on their returns.
Another thing AI can do that I can't...
I'm sure you could if you put your mind to it. It's not so bad, on peaceful mode. Just takes time.
Boring though without something to attack you imo
I actually like peace mode because I just focus on industrial optimization vs diverting my attention to attackers
Okay now this but applied towards the real world economy
Good to know it's good at something, I suppose? Sol seems like it always just wastes my time and writes terrible code.
The broligarchs are gunna fix that asap /s