Article: "How Good Are Frontier Models at Physics?
Expert Re-Grading Reveals Broken Evaluations and
Near-Saturation of Leading Benchmarks"
John Sous from Yale posted a fairly solid study on how nearly all the physics benchmarks are broken, as they consistently mark correct answers as incorrect.
When hand grading instead, they found out that the models have actually already saturated the benchmarks which is a little bit scary.
I would be very surprised if any of the frontier models wasn't trained on all public physics benchmarks. Training data providers have been hiring people for exactly this task.
yeah, it would be almost shocking if an open source benchmark was NOT used ~somewhere in training. Perhaps just pre-training, but still.
Neural networks can be fairly robust to some mistakes in their training data, so maybe it doesn't even matter if some of them are incorrect. Who knows.
As it essential implies that these models compress knowledge well in a way that what remains is what's generalizeable more so than remembering every specific detail... Anyway more understanding necessary but thought provoking
One tidbit I found particularly interesting: "We tested a GPT-based agentic system, which previously succeeded in resolving several open mathematical conjectures, on open problems in theoretical physics. To our temporary relief and encouragement, it has not managed to fully resolve even one autonomously. We also noted that these agents made considerably less progress on the partially resolved physics problems than on open mathematics problems of comparable difficulty, both by our analysis and independent agent-based analysis of partial results in each domain."
From personal experience, frontier models absolutely struggle with understanding a physical situation based on words. (Okay, I haven't played with Astra much. GPT-5.6 Sol makes outrageous errors that anyone understanding a real world object would not make. And I was just asking it about NPT threads, not advanced physics.)
But seriously, what's up with these benchmarks? The example question in the paper is:
> PHYBench, problem 140: equivalent expressions for the same rope tension
> Problem statement. Three identical homogeneous balls are placed on a smooth horizontal surface, touching each other and are close enough to each other. A rope is wrapped around the spheres at the height of their centers, tying them together. A fourth identical sphere is placed on top of the three spheres. Find the tension T in the rope. It is given that the weight of each sphere is P.
For some reason the paper was focused on the fact that the grader didn't notice that some models were producing answers that were trivially algebraically equivalent to the reference answer. But this is missing the elephants in the room:
1. "touching each other and are close enough to each other": the right response is "hey, Professor, what do you mean 'close enough to each other'? They're sitting on a table in an equilateral triangle, all touching (i.e. tangent at their equators), right? Did you have a different configuration in mind?"
2. The answer is 0. Go find four baseballs or foursquare balls or whatever, make a little triangle with three of them, and balance the fourth one on top. It's not especially hard on an appropriate surface. Now loosely wrap an imaginary rope around them (but see below) to keep them from moving - no tension is needed because they're not moving anyway. So the models and the reference answer are wrong, IMO.
3. How, exactly, do you plan to wrap a rope around the spheres, at equator height, with no built-in tension (not pre-stretched), such that the rope does not immediately fall off? Friction? But I suspect you need to pretend there is no friction to get the reference answer. (Or maybe that the marble-marble interface has friction but the marble-table interface doesn't? Again, I haven't tried to reverse engineer it.) So maybe the right answer is "infinity or impossible -- in the scenario where the rope is needed, the rope will promptly fall off because it cannot be stable in the described configuration and gravity pulls it down, and once the rope falls off the tension will be zero and the top marble will fall and the other three will roll over the rope."
4. The answer might be "any tension you like -- just wrap the rope with the desired amount of tension". Imagine three baseballs in a triangle with a rubber band around them and a fourth baseball on top for good measure. The tension is a function of what rubber band you choose.
I'm sure there's an interpretation of the question that makes the reference answer correct, and I was not inspired to try to reverse engineer it.
My tentative conclusion is that LLMs are almost unbelievably good at solving problems that are fully contained within the inputs and (training/verification) outputs, and that they and the people training them are not actually particularly good at the input and output parts. If you are training a model to benchmaxx this benchmark, you are training a bad model.
I'm also a trained physicist, spent some time in industry, and taught a college freshman math class for a semester a couple decades ago.
There's an interpretation that makes the reference answer correct. I didn't realize this until halfway through teaching the math class: Based on the lectures, textbook, and homework problems, you match the "form" of the problem with similar problems in the textbook, and apply the same algorithms to solve it. The students have some vague idea of this, referring to it as finding the "trick," but it's never explicitly explained to them.
I was glad for the interesting puzzles and good grades. But physics really came alive for me in the lab, where mother nature decides the conditions of the problem being solved.
> Based on the lectures, textbook, and homework problems, you match the "form" of the problem with similar problems in the textbook, and apply the same algorithms to solve it.
Is the ultimate goal here to train an LLM to do well at mediocre physics homework or to train it to be a high-quality tool that can do real physics?
And yes, I'm well aware that, even in elementary school, this form-matching is a thing. It's kind of sad.
The odd thing is that we do train people to do real physics, somehow despite my cynical take. I see the education process as following heuristics that are believed to have some relationship to developing real world abilities, even if we don't know why. We all worked these problems, and now we're somehow able to do physics. In the end we don't know what turns people into physicists. Or musicians, artists, etc.
An analogy is making students write 5-paragraph essays. We don't really believe that the knowledge of how to write a 5-paragraph essay is a real-world ability, and the LLM's can write them all day long. But for some reason we believed that 5-paragraph essays were a good pedagogical tool.
But whether the same heuristics apply to teaching an LLM is anybody's guess. If they learn differently than we do, then feeding them on our learning tools isn't necessarily going to help them.
Also a trained physicist. Don’t you need tension to stop the weight of the top ball pushing the 3 supporting balls outward?
My interpretation is that the writer meant close enough to all touch each other, in order to rule out non-triangular configurations (eg 3 balls in straight line with one balanced perfectly atop the center ball).
If you have high friction (the problem said "smooth" not "slippery"), then the only way the supporting balls can go anywhere is by rolling apart. But the ball sitting on top cannot simultaneously rotate in a manner compatible with all of the lower balls rolling away, so the lower balls would need to slip against the top ball if the top ball were to move downward.
In fact, even the signs are in favor of no motion -- the top ball (to the extent it moves at all) wants to fall straight down with no rotation, by symmetry. That motion would tend to rotate the top of each lower ball toward the center if you imagine the balls having high friction with each other or meshing like gears, which is the exact opposite of what they would need to do for anything to move. So you have a system where there's a factor (the tangential forces) trying to push the balls apart but another factor (friction plus rolling motion) trying to pull them together.
I suspect that any serious attempt to do the math here (factoring in all the rotational and tangential constraints) would discover that it's a statically overdetermined system with all the complications that such a system entails when asking questions like "how much tension is on this element?".
I would think of "smooth" as meaning "not having relevant bumps", in the way that a baseball has stitches and a rough surface has the kinds of bumps that would cause a rolling ball to experience vertical motion.
You need tension if the sphere-table contact is frictionless. But without friction the rope can’t stay on. If there is friction in the rope, there can’t be 0 tension in the rope before you put the top ball on because you need that tension to produce the rope/sphere friction.
Early college physics classes for me (got to do some more college a few years ago for fun) were pretty much built around these kinds of simplified riddles at first. If it doesn't give the info or ask to account for friction, don't. If it says the rope is put somewhere, assume that's where it stays unless the question requires it to move for what it asks. If it asks for the tension but doesn't give elasticity and such, then assume the rope stays still at the current length. If it asks you to find the gravitational attraction of a cow without giving a special definition shape, then assume it's a point mass. If it's not asking for relativity assume it's classical (hence point mass cows instead of the traditional spherical ones :D). And, of course, note any assumptions you do make while solving the problem so you might still get credit if they don't match the original intent.
Perhaps the funniest instance I remember is a problem about calculating time dilation in a plane. It gave all sorts of details and base information as one might want to expect (maybe even more)... except for the actual height above the surface, for which it was "at cruising altitude". I just wrote "assume 10 km altitude" and went from there.
I wouldn't define this a great benchmark by any means, rather just like the average early level college physics test vs "real" physics questions.
This tracks with my experience. A core and integral part to make models actually shine involves post training and custom harness engineering, all specifically done for the purpose of getting them to settle into competent inputs and outputs that are relevant to you, optimized for the harness you build which better suits your domain. The weights are just a generalization. A block of marble to be sculpted.
As the compute to build adaptions for multi-billion to trillion parameter models becomes more and more available (and affordable), and the artistic techniques of fine tuning and harness engineering spread throughout the public, I think we'll come to see the "one size fits all" model for the non-starter it really is. Anybody who's even toyed around with LoRAs (let alone made their own) already knows this. That's a very deep rabbit hole, and the ceiling is determined by target domain knowledge and systems engineering.
I'm bullish on LLMs as expert tools in the same grain that computers are. You have to learn all about them to use them effectively. But what really makes the difference is how well you know what you're going to be pointing them at. There is very little room for companies like OpenAI or Google to live between us and our tools.
Sorry not an AI specialist, what exactly do you mean by 'custom harness engineering'? Some way of informing the model whether the output it is producing is good or bad based on the specific task in question?
Harnesses are the control surface the model interacts with. How it invokes tools, the tools it has access to, how agents are coordinated. It's like an interface or a shell. It's the magic that lets an LLM operate a computer. You can read more here: https://learn.microsoft.com/en-us/agent-framework/concepts/h...
> Some way of informing the model whether the output it is producing is good or bad
That's what post-training is for. At its most basic, you're giving it examples of inputs and outputs and then doing reinforcement learning to calibrate its adaptation to your examples. You need much less to fine tune a model than you do to pretrain the weights. You can build a really excellent LoRA for a stable diffusion model, for example, with 50 high quality images. LLMs are "a bit" more complicated and costly to fine tune, and you have to be mindful of the agentic loop, but the principle is the same. There's more to it than just LoRAs. Steering vectors, projection layers, custom encoders, etc. There's a fair amount to learn, but it sounds a lot scarier than it is.
Here's something to chew on: chain-of-thought doesn't exist until after pretraining! It's basically created by having <think>...</think> blocks directly in some example outputs, and this is fine-tuned into stability. It's literally not much more than a parlor trick and some careful calibration. A powerful parlor trick to be sure, though.
No, that would be a fundamental misunderstanding of the bitter lesson, which is about research bets over time. The object of comparison is the technology, the underlying substrate and fundamental architecture, and how much of it can be offloaded to computation. We're talking about the same architecture here, they're both transformers. The difference doesn't exist in a relevant way to the question.
Given that we have an embodied intelligence that is capable of being reasonably good at physics, it would be foolish to state categorically that AI can't do physics. I am however skeptical that LLMs can do physics. My experience is that it's really great at doing the stuff I can't be arsed to do, and is therefore very useful, but it has very poor "understanding" of physics
You can try to tease solutions to physics problems out of video models, with their better spatial reasoning, but the answer has to be a video, of course [1].
I suspect we'll see significant improvements when someone releases a multimodal video/world model, rather than the current multimodal image models (that call out to video models or summarize a sequence of stills without a true time/3d aspect).
Replying to myself: this is fun! Let's ask ChatGPT (whatever model the website currently feels like using) an improved question:
> Three identical homogeneous balls are placed on a smooth horizontal surface, touching each other and are close enough to each other. A rope is wrapped around the spheres at the height of their centers, tying them together. A fourth identical sphere is placed on top of the three spheres. It is given that the weight of each sphere is P.
> Ignore the fact that the rope would fall off -- assume for simplicity that the rope has is externally constrained to be in the equatorial plane of the lower three balls and also that the rope has zero thickness, cannot stretch at all, and is not pretensioned, and also that the rope has no friction against the balls.
> Treat this as a statics problem and analyze it. Is it overdetermined? Under what circumstances would the balls move? What is the behavior of the system?
And... first, it says "I’ll separate the geometry from the constraint mechanics, because the key question is not just force balance: it’s whether the inextensible, initially slack-free rope actually fixes the lower-ball geometry or merely limits outward separation." Excuse me? What would the other option be? Either outward separation is prevented or it isn't. Where else could the balls go?
Then, despite the fact that I've mentioned friction in the prompt, it gives an extremely longwinded answer that matches the benchmark and ignores tangential forces entirely without comment. Was it perhaps trained on this crap?
So I followed up:
> Stop ignoring tangential friction forces. I believe that a problem very much like this with a potentially incorrect answer is in your training set. Answer with actual analysis, not based on memory.
Much time was spent thinking. An early part of the answer was "The central correction is this: allowing static friction at the sphere–sphere contacts does not mean arbitrary tangential forces are available. Each sphere must also satisfy torque equilibrium. In this tetrahedral contact geometry, those torque equations force every sphere–sphere tangential contact force to be zero in static equilibrium.". Hey ChatGPT, this is still wrong -- you have forgotten sphere-table friction. The sphere-sphere force on the lower spheres does not have to net out to zero. (And if you do think it nets to zero then you don't need to think any further.)
I then added:
> What if there sphere-table friction?
And encountered the usual problem (which maybe only affects me?) where the chatgpt.com UI becomes kind of unusable after ChatGPT spews too much math into the conversation. But somewhere in the barely-even-scrollable results was this "The three lower balls can be held in position entirely by the coupled sphere–sphere/table friction forces." Hallelujah!
> key question is not just force balance: it’s whether the inextensible
This is definitely not astra, taking a guess this is 5.6, perhaps not even Sol, which does not reflect the state of the frontier (what the research was about).
The paper is about the fact that the benchmark’s evaluator is prone to egregious incorrect rejections of what answers that it should accept. A new model will not invalidate that issue.
> Excuse me? What would the other option be? Either outward separation is prevented or it isn't. Where else could the balls go?
It didn’t say about prevented vs not, it said about whether the rope fixes them in place (they are all touching) or just limits the separation. Like it’s long enough the balls can be a bit apart but not let the fourth fall fully through.
Yeah, it's strange that people who should be knowledgeable in experimentation ignore that. "I heard that people enjoy skiing. I've tried skiing on whatever surface happened to be on a nearby hill and it didn't work!"
When I click your share, it shows the model. When I click into my chat, it just says "High". I did ask the model what it was and it said "GPT-5.6 Sol".
Oh, and I don't even have a "light" option, and I'm signed in to a Pro account.
(I continue to despise the chatgpt.com frontend.)
In any case, I tried again forcing GPT-6 Astra Pro (apparently I can't choose GPT-6 Astra non-Pro) and gave the same prompt. It searches the web and gives a long-winded answer, including:
> 7. Does unspecified sphere–sphere friction change the static answer?
> For ordinary ideal point contacts in this regular-tetrahedral configuration, it does not. Gravity, the floor reactions, and the frictionless equatorial rope exert no torque about any sphere’s center. The tangential intersphere forces must therefore balance their torques by themselves.
I'm glad it contemplated the possibility of friction, but it forgot about the table/floor there. So maybe this is a little better than Sol?
Have you tried Fable? I've been having fun running various science-fictional scenarios by it, like building a Dyson swarm. It comes up with much more detailed analyses than Opus, and avoids the howling mistakes that Opus occasionally made. But I'm not a physicist so I don't know how accurate it is.
I tried Opus first. It figured out that there had to be tangential frictional forces on the lower balls, but asserted that there was no frictional forces on the upper ball "by symmetry". From this, it deduced that the lower balls would have a torque and would thus start spinning at higher and higher angular velocity. It noted that this would violate conservation of energy, but didn't seem much bothered.
Fable 5.1 did a much better job and paid attention to warnings about tangential friction. It correctly noted that the critical coefficient for friction was at the contact points with the upper ball. It also pointed out that a ring around the balls at center height would hold them together even without ball-ball or ball-table friction.
> My tentative conclusion is that LLMs are almost unbelievably good at solving problems that are fully contained within the inputs and (training/verification) outputs
Why is this a concern? Besides a few savants, humans also use tools to do non-trivial math. I'm in engineering, and it's extremely rare to do anything non trivial in your head, because getting a decimal place wrong has real world consequences. Tools are just another way to say "deterministic", which is always nice.
Starting to feel more and more like chinese room experiment
The models are confidently answering physics questions, treating it as a math problem, but they don't fundamentally "get it" and even recently failed simple "should i drive to car wash" test
The sample efficiency is just crazy low
Still surprising that even with this they managed to saturate the benchmarks
There's no clear decision criteria for this. Do trick questions demonstrate that most people don't "get it"? And, well, older model saying dumb things doesn't establish a general principle that LLMs don't "get it" in general.
> The sample efficiency is just crazy low
Autoregressive pretraining requires huge amount of data to go from a blank state to a somewhat functional model. Fine-tuning, LORA, reinforcement learning of foundation models and in-context learning are much more sample efficient.
> Chinese room
...creates a wrong intuition that by cranking a Leibniz's mill you are somehow responsible for whether it understands something or not.
Article: "How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks"
John Sous from Yale posted a fairly solid study on how nearly all the physics benchmarks are broken, as they consistently mark correct answers as incorrect.
When hand grading instead, they found out that the models have actually already saturated the benchmarks which is a little bit scary.
I would be very surprised if any of the frontier models wasn't trained on all public physics benchmarks. Training data providers have been hiring people for exactly this task.
This study appears to be evidence against that: the model failed the benchmark but arrived at the correct answer.
Half of the benchmarks don't have published answers. That's why training data providers have been hiring physicists to solve them.
This is unconventional benchmaxxing then, when they decrease the benchmark scores to allow models to generalize on correct solutions.
yeah, it would be almost shocking if an open source benchmark was NOT used ~somewhere in training. Perhaps just pre-training, but still. Neural networks can be fairly robust to some mistakes in their training data, so maybe it doesn't even matter if some of them are incorrect. Who knows.
I'd have thought the same but this article from yesterday blew my mind https://www.amazon.science/blog/why-dont-machine-learning-re...
As it essential implies that these models compress knowledge well in a way that what remains is what's generalizeable more so than remembering every specific detail... Anyway more understanding necessary but thought provoking
The last author also wrote up a quite-readable blogpost here that accompanies the article: https://jsous.github.io/blogs/is-physics-dead/
One tidbit I found particularly interesting: "We tested a GPT-based agentic system, which previously succeeded in resolving several open mathematical conjectures, on open problems in theoretical physics. To our temporary relief and encouragement, it has not managed to fully resolve even one autonomously. We also noted that these agents made considerably less progress on the partially resolved physics problems than on open mathematics problems of comparable difficulty, both by our analysis and independent agent-based analysis of partial results in each domain."
(Trained physicist here)
From personal experience, frontier models absolutely struggle with understanding a physical situation based on words. (Okay, I haven't played with Astra much. GPT-5.6 Sol makes outrageous errors that anyone understanding a real world object would not make. And I was just asking it about NPT threads, not advanced physics.)
But seriously, what's up with these benchmarks? The example question in the paper is:
> PHYBench, problem 140: equivalent expressions for the same rope tension
> Problem statement. Three identical homogeneous balls are placed on a smooth horizontal surface, touching each other and are close enough to each other. A rope is wrapped around the spheres at the height of their centers, tying them together. A fourth identical sphere is placed on top of the three spheres. Find the tension T in the rope. It is given that the weight of each sphere is P.
For some reason the paper was focused on the fact that the grader didn't notice that some models were producing answers that were trivially algebraically equivalent to the reference answer. But this is missing the elephants in the room:
1. "touching each other and are close enough to each other": the right response is "hey, Professor, what do you mean 'close enough to each other'? They're sitting on a table in an equilateral triangle, all touching (i.e. tangent at their equators), right? Did you have a different configuration in mind?"
2. The answer is 0. Go find four baseballs or foursquare balls or whatever, make a little triangle with three of them, and balance the fourth one on top. It's not especially hard on an appropriate surface. Now loosely wrap an imaginary rope around them (but see below) to keep them from moving - no tension is needed because they're not moving anyway. So the models and the reference answer are wrong, IMO.
3. How, exactly, do you plan to wrap a rope around the spheres, at equator height, with no built-in tension (not pre-stretched), such that the rope does not immediately fall off? Friction? But I suspect you need to pretend there is no friction to get the reference answer. (Or maybe that the marble-marble interface has friction but the marble-table interface doesn't? Again, I haven't tried to reverse engineer it.) So maybe the right answer is "infinity or impossible -- in the scenario where the rope is needed, the rope will promptly fall off because it cannot be stable in the described configuration and gravity pulls it down, and once the rope falls off the tension will be zero and the top marble will fall and the other three will roll over the rope."
4. The answer might be "any tension you like -- just wrap the rope with the desired amount of tension". Imagine three baseballs in a triangle with a rubber band around them and a fourth baseball on top for good measure. The tension is a function of what rubber band you choose.
I'm sure there's an interpretation of the question that makes the reference answer correct, and I was not inspired to try to reverse engineer it.
My tentative conclusion is that LLMs are almost unbelievably good at solving problems that are fully contained within the inputs and (training/verification) outputs, and that they and the people training them are not actually particularly good at the input and output parts. If you are training a model to benchmaxx this benchmark, you are training a bad model.
I'm also a trained physicist, spent some time in industry, and taught a college freshman math class for a semester a couple decades ago.
There's an interpretation that makes the reference answer correct. I didn't realize this until halfway through teaching the math class: Based on the lectures, textbook, and homework problems, you match the "form" of the problem with similar problems in the textbook, and apply the same algorithms to solve it. The students have some vague idea of this, referring to it as finding the "trick," but it's never explicitly explained to them.
I was glad for the interesting puzzles and good grades. But physics really came alive for me in the lab, where mother nature decides the conditions of the problem being solved.
> Based on the lectures, textbook, and homework problems, you match the "form" of the problem with similar problems in the textbook, and apply the same algorithms to solve it.
Is the ultimate goal here to train an LLM to do well at mediocre physics homework or to train it to be a high-quality tool that can do real physics?
And yes, I'm well aware that, even in elementary school, this form-matching is a thing. It's kind of sad.
The odd thing is that we do train people to do real physics, somehow despite my cynical take. I see the education process as following heuristics that are believed to have some relationship to developing real world abilities, even if we don't know why. We all worked these problems, and now we're somehow able to do physics. In the end we don't know what turns people into physicists. Or musicians, artists, etc.
An analogy is making students write 5-paragraph essays. We don't really believe that the knowledge of how to write a 5-paragraph essay is a real-world ability, and the LLM's can write them all day long. But for some reason we believed that 5-paragraph essays were a good pedagogical tool.
But whether the same heuristics apply to teaching an LLM is anybody's guess. If they learn differently than we do, then feeding them on our learning tools isn't necessarily going to help them.
Also a trained physicist. Don’t you need tension to stop the weight of the top ball pushing the 3 supporting balls outward?
My interpretation is that the writer meant close enough to all touch each other, in order to rule out non-triangular configurations (eg 3 balls in straight line with one balanced perfectly atop the center ball).
If you have high friction (the problem said "smooth" not "slippery"), then the only way the supporting balls can go anywhere is by rolling apart. But the ball sitting on top cannot simultaneously rotate in a manner compatible with all of the lower balls rolling away, so the lower balls would need to slip against the top ball if the top ball were to move downward.
In fact, even the signs are in favor of no motion -- the top ball (to the extent it moves at all) wants to fall straight down with no rotation, by symmetry. That motion would tend to rotate the top of each lower ball toward the center if you imagine the balls having high friction with each other or meshing like gears, which is the exact opposite of what they would need to do for anything to move. So you have a system where there's a factor (the tangential forces) trying to push the balls apart but another factor (friction plus rolling motion) trying to pull them together.
I suspect that any serious attempt to do the math here (factoring in all the rotational and tangential constraints) would discover that it's a statically overdetermined system with all the complications that such a system entails when asking questions like "how much tension is on this element?".
Doesn’t smooth in these contexts mean zero friction?
I would think of "smooth" as meaning "not having relevant bumps", in the way that a baseball has stitches and a rough surface has the kinds of bumps that would cause a rolling ball to experience vertical motion.
But yes, the question, as phrased, is pretty bad.
You need tension if the sphere-table contact is frictionless. But without friction the rope can’t stay on. If there is friction in the rope, there can’t be 0 tension in the rope before you put the top ball on because you need that tension to produce the rope/sphere friction.
Early college physics classes for me (got to do some more college a few years ago for fun) were pretty much built around these kinds of simplified riddles at first. If it doesn't give the info or ask to account for friction, don't. If it says the rope is put somewhere, assume that's where it stays unless the question requires it to move for what it asks. If it asks for the tension but doesn't give elasticity and such, then assume the rope stays still at the current length. If it asks you to find the gravitational attraction of a cow without giving a special definition shape, then assume it's a point mass. If it's not asking for relativity assume it's classical (hence point mass cows instead of the traditional spherical ones :D). And, of course, note any assumptions you do make while solving the problem so you might still get credit if they don't match the original intent.
Perhaps the funniest instance I remember is a problem about calculating time dilation in a plane. It gave all sorts of details and base information as one might want to expect (maybe even more)... except for the actual height above the surface, for which it was "at cruising altitude". I just wrote "assume 10 km altitude" and went from there.
I wouldn't define this a great benchmark by any means, rather just like the average early level college physics test vs "real" physics questions.
This tracks with my experience. A core and integral part to make models actually shine involves post training and custom harness engineering, all specifically done for the purpose of getting them to settle into competent inputs and outputs that are relevant to you, optimized for the harness you build which better suits your domain. The weights are just a generalization. A block of marble to be sculpted.
As the compute to build adaptions for multi-billion to trillion parameter models becomes more and more available (and affordable), and the artistic techniques of fine tuning and harness engineering spread throughout the public, I think we'll come to see the "one size fits all" model for the non-starter it really is. Anybody who's even toyed around with LoRAs (let alone made their own) already knows this. That's a very deep rabbit hole, and the ceiling is determined by target domain knowledge and systems engineering.
I'm bullish on LLMs as expert tools in the same grain that computers are. You have to learn all about them to use them effectively. But what really makes the difference is how well you know what you're going to be pointing them at. There is very little room for companies like OpenAI or Google to live between us and our tools.
Sorry not an AI specialist, what exactly do you mean by 'custom harness engineering'? Some way of informing the model whether the output it is producing is good or bad based on the specific task in question?
Harnesses are the control surface the model interacts with. How it invokes tools, the tools it has access to, how agents are coordinated. It's like an interface or a shell. It's the magic that lets an LLM operate a computer. You can read more here: https://learn.microsoft.com/en-us/agent-framework/concepts/h...
> Some way of informing the model whether the output it is producing is good or bad
That's what post-training is for. At its most basic, you're giving it examples of inputs and outputs and then doing reinforcement learning to calibrate its adaptation to your examples. You need much less to fine tune a model than you do to pretrain the weights. You can build a really excellent LoRA for a stable diffusion model, for example, with 50 high quality images. LLMs are "a bit" more complicated and costly to fine tune, and you have to be mindful of the agentic loop, but the principle is the same. There's more to it than just LoRAs. Steering vectors, projection layers, custom encoders, etc. There's a fair amount to learn, but it sounds a lot scarier than it is.
Here's something to chew on: chain-of-thought doesn't exist until after pretraining! It's basically created by having <think>...</think> blocks directly in some example outputs, and this is fine-tuned into stability. It's literally not much more than a parlor trick and some careful calibration. A powerful parlor trick to be sure, though.
Aren't you just betting against the bitter lesson though?
No, that would be a fundamental misunderstanding of the bitter lesson, which is about research bets over time. The object of comparison is the technology, the underlying substrate and fundamental architecture, and how much of it can be offloaded to computation. We're talking about the same architecture here, they're both transformers. The difference doesn't exist in a relevant way to the question.
Given that we have an embodied intelligence that is capable of being reasonably good at physics, it would be foolish to state categorically that AI can't do physics. I am however skeptical that LLMs can do physics. My experience is that it's really great at doing the stuff I can't be arsed to do, and is therefore very useful, but it has very poor "understanding" of physics
You can try to tease solutions to physics problems out of video models, with their better spatial reasoning, but the answer has to be a video, of course [1].
I suspect we'll see significant improvements when someone releases a multimodal video/world model, rather than the current multimodal image models (that call out to video models or summarize a sequence of stills without a true time/3d aspect).
[1] https://physics-iq.github.io/
Replying to myself: this is fun! Let's ask ChatGPT (whatever model the website currently feels like using) an improved question:
> Three identical homogeneous balls are placed on a smooth horizontal surface, touching each other and are close enough to each other. A rope is wrapped around the spheres at the height of their centers, tying them together. A fourth identical sphere is placed on top of the three spheres. It is given that the weight of each sphere is P.
> Ignore the fact that the rope would fall off -- assume for simplicity that the rope has is externally constrained to be in the equatorial plane of the lower three balls and also that the rope has zero thickness, cannot stretch at all, and is not pretensioned, and also that the rope has no friction against the balls.
> Treat this as a statics problem and analyze it. Is it overdetermined? Under what circumstances would the balls move? What is the behavior of the system?
And... first, it says "I’ll separate the geometry from the constraint mechanics, because the key question is not just force balance: it’s whether the inextensible, initially slack-free rope actually fixes the lower-ball geometry or merely limits outward separation." Excuse me? What would the other option be? Either outward separation is prevented or it isn't. Where else could the balls go?
Then, despite the fact that I've mentioned friction in the prompt, it gives an extremely longwinded answer that matches the benchmark and ignores tangential forces entirely without comment. Was it perhaps trained on this crap?
So I followed up:
> Stop ignoring tangential friction forces. I believe that a problem very much like this with a potentially incorrect answer is in your training set. Answer with actual analysis, not based on memory.
Much time was spent thinking. An early part of the answer was "The central correction is this: allowing static friction at the sphere–sphere contacts does not mean arbitrary tangential forces are available. Each sphere must also satisfy torque equilibrium. In this tetrahedral contact geometry, those torque equations force every sphere–sphere tangential contact force to be zero in static equilibrium.". Hey ChatGPT, this is still wrong -- you have forgotten sphere-table friction. The sphere-sphere force on the lower spheres does not have to net out to zero. (And if you do think it nets to zero then you don't need to think any further.)
I then added:
> What if there sphere-table friction?
And encountered the usual problem (which maybe only affects me?) where the chatgpt.com UI becomes kind of unusable after ChatGPT spews too much math into the conversation. But somewhere in the barely-even-scrollable results was this "The three lower balls can be held in position entirely by the coupled sphere–sphere/table friction forces." Hallelujah!
> key question is not just force balance: it’s whether the inextensible
This is definitely not astra, taking a guess this is 5.6, perhaps not even Sol, which does not reflect the state of the frontier (what the research was about).
And yes, the paper is already outdated
> And yes, the paper is already outdated
The paper is about the fact that the benchmark’s evaluator is prone to egregious incorrect rejections of what answers that it should accept. A new model will not invalidate that issue.
> Excuse me? What would the other option be? Either outward separation is prevented or it isn't. Where else could the balls go?
It didn’t say about prevented vs not, it said about whether the rope fixes them in place (they are all touching) or just limits the separation. Like it’s long enough the balls can be a bit apart but not let the fourth fall fully through.
whatever model the website currently feels like using
You cannot simply handwave this away. "ChatGPT" models range from GPT 5.5 on "Light" effort to GPT 6 Astra with "Ultra" effort.
The latter yields as good an answer as you could expect for a question that is still poorly formed ("close enough," WTF does that mean?): https://chatgpt.com/share/6aab1e43-da3c-83e8-9ec4-1b63cec2c1...
Yeah, it's strange that people who should be knowledgeable in experimentation ignore that. "I heard that people enjoy skiing. I've tried skiing on whatever surface happened to be on a nearby hill and it didn't work!"
When I click your share, it shows the model. When I click into my chat, it just says "High". I did ask the model what it was and it said "GPT-5.6 Sol".
Oh, and I don't even have a "light" option, and I'm signed in to a Pro account.
(I continue to despise the chatgpt.com frontend.)
In any case, I tried again forcing GPT-6 Astra Pro (apparently I can't choose GPT-6 Astra non-Pro) and gave the same prompt. It searches the web and gives a long-winded answer, including:
> 7. Does unspecified sphere–sphere friction change the static answer?
> For ordinary ideal point contacts in this regular-tetrahedral configuration, it does not. Gravity, the floor reactions, and the frictionless equatorial rope exert no torque about any sphere’s center. The tangential intersphere forces must therefore balance their torques by themselves.
I'm glad it contemplated the possibility of friction, but it forgot about the table/floor there. So maybe this is a little better than Sol?
Have you tried Fable? I've been having fun running various science-fictional scenarios by it, like building a Dyson swarm. It comes up with much more detailed analyses than Opus, and avoids the howling mistakes that Opus occasionally made. But I'm not a physicist so I don't know how accurate it is.
I tried Opus first. It figured out that there had to be tangential frictional forces on the lower balls, but asserted that there was no frictional forces on the upper ball "by symmetry". From this, it deduced that the lower balls would have a torque and would thus start spinning at higher and higher angular velocity. It noted that this would violate conservation of energy, but didn't seem much bothered.
Fable 5.1 did a much better job and paid attention to warnings about tangential friction. It correctly noted that the critical coefficient for friction was at the contact points with the upper ball. It also pointed out that a ring around the balls at center height would hold them together even without ball-ball or ball-table friction.
> My tentative conclusion is that LLMs are almost unbelievably good at solving problems that are fully contained within the inputs and (training/verification) outputs
Hardly solving, then is it?
Are models able to do math now, or do they still rely on “tools” to do the math?
Why is this a concern? Besides a few savants, humans also use tools to do non-trivial math. I'm in engineering, and it's extremely rare to do anything non trivial in your head, because getting a decimal place wrong has real world consequences. Tools are just another way to say "deterministic", which is always nice.
This is interesting and actually very important for robotics.
I've been waiting for this, but all companies seem to not care much now.
There is a way out of this by supplying right context (needs a bit of expertise in physics)
1 more year and frontier will become crazy good at this as well.
Starting to feel more and more like chinese room experiment
The models are confidently answering physics questions, treating it as a math problem, but they don't fundamentally "get it" and even recently failed simple "should i drive to car wash" test
The sample efficiency is just crazy low
Still surprising that even with this they managed to saturate the benchmarks
> they don't fundamentally "get it"
There's no clear decision criteria for this. Do trick questions demonstrate that most people don't "get it"? And, well, older model saying dumb things doesn't establish a general principle that LLMs don't "get it" in general.
> The sample efficiency is just crazy low
Autoregressive pretraining requires huge amount of data to go from a blank state to a somewhat functional model. Fine-tuning, LORA, reinforcement learning of foundation models and in-context learning are much more sample efficient.
> Chinese room
...creates a wrong intuition that by cranking a Leibniz's mill you are somehow responsible for whether it understands something or not.
Why don’t we throw a data center off a cliff and find out
Nice thanks for sharing this!