> we ask them to stop testing advanced mathematical problems on proprietary models.
Maybe I'm alone on this, but for some reason these sorts of requests strike me as akin to gatekeeping how someone should breathe air. It's math... the numbers and symbols are just out there in the platonic realm available for anyone to do as they like with them. It's patently absurd to request other people to stop.
Ensuring credit where credit is due? That's fine. If your model incorporates the efforts of many others, then it's reasonable to request acknowledgement of everyone who contributed (even indirectly). But that's not what the request states — presumably their ask subsumes any advanced ML model, including those that weren't trained on a giant corpus of text.
> akin to gatekeeping how someone should breathe air.
The law is also a sequence of letters available to anyone. Yet you cannot practice without passing the bar exam. And that is not the only example; many professions can only be practiced by licensed individuals: medical doctors, journalists, electricians...
To quote Noam Brown, "Our main focus is shipping great models so everyone can use them to make discoveries of their own" [1]. Spending 15 million dollars of compute with an internal model to blitz and scoop a resolution of Navier-Stokes that someone is already on track to resolve (or, from my understanding of why OpenAI did this, had already resolved, as staff at OpenAI stated their attempt was prompted by rumors of a resolution by Anthropic) is in complete opposition to this stated claim. That act is what this portion of the recommendation is in response to: OpenAI perpetually holding out internal models and tooling, using them to solve important problems in math, and thus themselves holding a monopoly on certain aspects of mathematics. Why is it absurd to ask people who are in a position to do an obviously damaging act to not do so, especially when those same people were the ones who asked for your advice in the first place and claimed to not want to do said act previously?
>was prompted by rumors of a resolution by Anthropic
reminds an old sci-fi story where top engineers were shown a video of a genius inventor who invented anti-gravity and unfortunately died while testing his apparatus which was clearly anti-gravitating in the video. Thus believing that it is a solved problem, just need to rediscover the lost solution, the engineers quickly developed anti-gravity. The original video happened to be a fake specially created for that purpose.
> akin to gatekeeping how someone should breathe air.
Yes, and professions that do that - that vocally insist there is an essential human element to the craft - will likely fare better than the ones that say "whatever, code is code" or "whatever, math is math".
This is smart. We might not like it, but SWEs are actually the dumb ones with their utilitarian attitudes to AI. We're digging our own grave. Meanwhile, professions such as writers or musicians are positioning themselves better by shunning "artists" that simply pull the lever of a gen AI system.
The problem with proprietary models is that you don't get to poke inside and see what it is doing and how it arrives at its answer, which is precisely what mathematicians do. Mathematical understanding derives less from any particular result than the insights and methods that pave the road to results. Whenever a theorem is proved, researchers seek to unpack the proof and get inside the author's mind to learn their ways of thinking.
LLM generated results might benefit mathematical understanding if people can inspect their intermediate reasoning traces to discover erroneous human biases or patterns that they might have previously overlooked. Otherwise, the results might as well be produced by oracles.
> The problem with proprietary models is that you don't get to poke inside and see what it is doing and how it arrives at its answer, which is precisely what mathematicians do
Uh, a hypothetical fully open source model would have the exact same problem, because LLMs rely on emergent phenomena and no one understands why they work.
>Otherwise, the results might as well be produced by oracles.
no. The math result is a result only when it includes proof. The proof is the value here. The way somebody came to it isn't really important - we don't know how Newton came to his results, whether it was apple or pear, and it isn't really important. Or how Einstein was walking the city streets looking at the tower watches - it is just historic curiosity having no real value for science.
That has been one of the greatest thing about math departments - smooth talkers were always clearly visible as smooth talkers. You're either producing proofs, or you're anything but a mathematician.
I feel for mathematicians. They have similar situation like we have in programming. Well, we all just have to evolve and adjust (in particular reign in our pride as just in a few years - i think once LLMs start hitting 100T+ - we may loose our "top of God's creation" position). Any attempts at gatekeeping, ludditing, organizing in quasi observational/advisory boards really intended to protect their tenures, etc. ... - well, you just can't stop the wave.
It all reminds how Catholic Church insisted on responsible release of the Bible in German. The Church even unleashed the devastating 30 Years War trying to protect its monopoly on religion including the right to sell indulgences, etc.
>AI labs should provide significant support, including funding
And now all those "responsible math" and advisory boards would like to preserve their monopoly on math and would like to sell the indulgences to the AI labs. As usually it is all about money and power, not about science. As a Math PhD dropout myself i feel a bit of a shame and disappointment for that undignified scramble by the mathematics establishment. Being smart they should have led the way and show an example to the rest of humanity ...
This is exactly what I was thinking as I read it, along with the bit that AI labs should support human understanding. The whole thing smells as they're trying to place this burden on labs that are just offering a tokens service; them publishing about particular topics is essentially a side quest in the first place IMO. If a community wants to create Math labs dedicated to understanding AI discoveries in the field, then they're free. If they want to petition AI labs for financial support, they're also free. But this wording where they're trying to dictate what AI labs should do (outside of their primary business) just smells.
But on one hand there are labs that pushes tens of millions $ to mine for publicity, and on one hand mostly underfunded researchers trying to improve general understanding. Big ai may seriously harm math and when Pr value diminishes down nobody is there to keep pushing.
“researchers trying to improve general understanding”
Pretty sure AI will do better for that.. mathematicians need to be centaurs like the rest of us and stop rhetoric that is going to make existing math centaurs feel like they might get math-cancelled
My point is that there is a real difference between massive ai company budgets and research mathematicians driven it, it is not an argument against use of ai which seems inevitable.
>mathematicians need to be centaurs like the rest of us and stop rhetoric that is going to make existing math centaurs feel like they might get math-cancelled
I don't see why mathematicians should be protected from AI anymore than any other profession. It's either everybody or nobody, not fair on the face of it otherwise.
TFA isn't asking for mathematicians to be protected from AI. It's asking AI labs to hold themselves to the standards of the mathematical community:
- releasing papers using the normal process to allow peer review
- giving talks etc to disseminate knowledge so humans understand the result
- writing papers in a way (standard terminology etc) that allows mathematicians to digest the result (some AI math papers comprise a huge verbose load of non-standard terminology and waffle and then a massive lean proof. This is very hard for humans to actually understand, and means it's hard for others to take the work forward.)
- giving appropriate credit to results that are used to derive the work
It includes some specific recommendations for situations where the person prompting the model is not in a position to understand the output, and frankly these are really welcome given situations like the recent case at Anthropic where a non-mathematician at Anthropic prompted claude to make a significant improvement to the bounds of a problem related to the Riemann Zeta function[1] which led to widespread misreporting and claims (not by Anthropic themselves notably) that the Riemann hypothesis itself had been proved, which is emphatically not the case.
Research mathematics is fundamentally a collaborative activity and the way in which some of these results are released is done to maximise PR but means a ton of the mathematical value is left on the table.
[1] https://www.anthropic.com/research/riemann-zeta. As I understand it, the Riemann Hypothesis says that all non-trivial zeroes of the zeta function lie on a line called the critical line. Two centuries of previous work had established that at least something like 40.9% of the zeroes lie on the line and noone has ever found a non-trivial zero that does not lie on that line. Claude (with prompting from a non-mathematician to "try harder" etc) improved this bound massively to 67%. Now a lot of people said things like "OK so all we've got to do is to improve that to 100% and we've proved the RH", which is definitely not true unfortunately, because you can say that in the limit the proportion of the zeroes on the line is 100% and still have infinitely many which are not.
we do not endorse this practice, and we ask them to stop testing advanced mathematical problems on proprietary models
From point 2 under section "1. Background":
AI labs should provide significant support, including funding, to help develop this understanding
Point 3 under the same section:
The development of human understanding must remain organic and community led. It should not be directed by AI labs, even when the labs have produced the results.
If, as you say, they are "asking AI labs to hold themselves to the standards of the mathematical community", I must conclude that the mathematical community
1. Ideally wants a monopoly on mathematics research.
2. Demands money from those who dare violate their ideal monopoly.
3. Insists that the ideal monopoly remain in charge.
Math is a tool. It's what people do with the tools after they are invented that matters - and with mathematical tools sometimes that wait is hundreds of years.
The mathematics community has found that the tools have value long after being invented if the inventor leaves his tools in a specific format. The community is doing its best to preserve that - not because they want less toolmakers, but rather because they want the tools to be useful when they become needed.
Maintaining the existing standards is in fact a form of protection from AI disruption. They are asking the AI companies to follow their norms, instead of them having to conform to new norms created by AI. They don’t want to have to change the way they do things, understandably!, and are asking the companies to accommodate their way of life.
Mathematician was never really a "profession" like the others. It doesn't pay well and is largely confined to academia. If you're really good and want to get paid, you don't do the kinds of problems AI have been taking a crack at. You go to a quant firm or some tech company where this kinda math actually matters once in a blue moon
Air is there for all to breathe. It's a more-or-less fungible, free resource for all to use and the consumption of it is a basic requirement for life.
The AI companies, in contrast are using vast financial, human and compute resources to train and operate specialised models that are not available to the public. The advisory group's job is to give non-binding advice on how they can use this privately owned technology in a responsible manner that avoids doing unnecessary harm to the mathematical community. To call that gatekeeping misses the point entirely.
The very first paragraph of the article says (emphasis mine):
> At present, some frontier AI labs are testing advanced mathematical problems on proprietary models that remain inaccessible to the broader scientific community. Our recommendations are formulated with this practical context in mind.
The Navier Stokes proof came from an internal model that AFAIK still has not been released even in a limited way to scientists, let alone to the general public. Publicly available models are not what these mathematicians are talking about.
> gatekeeping how someone should breathe air. It's math... the numbers and symbols are just out there in the platonic realm available for anyone to do as they like with them. It's patently absurd to request other people to stop.
The chemical compounds of all kinds are also out there, available for anyone to do as they like with them. Say, mixing ammonium chlorate with peroxide, why not? Or potassium permanganate with powdered aluminum. It's patently absurd to request other people to stop.
> The chemical compounds of all kinds are also out there,
1. Lol but they're literally not
2. A sample of chemical compound and a piece of math don't share literally any ontological qualities - you might as well have tried to make a comparison between math and nude pictures of the President
> you might as well have tried to make a comparison between math and nude pictures of the President.
The latter is but a very large number, interpreted in a particular way. Have you heard of "illegal numbers"? Yeah, apparently they exist. That damn state, treading on literally everything that humans might do as if it's any of its business.
Mostly pretty straight forward. The idea that proofs be desloppified, attribute existing literature properly, and published somewhere expediently where it can be commented on, with artifacts for verification, is all very uncontroversial stuff.
I think the spicy take is definitely this stance that longstanding mathematical problems shouldn't be used as benchmarks for (specifically proprietary) models. Stated right at the very top.
The justification is pretty clear.
> The use of proprietary internal models by AI labs to do mathematical research risks creating a two-tier system where labs outrun the rest of the field, effectively alienating the mathematical community from its own discipline.
It is all fun and games (for non mathematicians) when mathematicians can't compete with AI labs but I think the more dangerous direction is when this starts being true for the rest of everything. For example cybersecurity or whatnot. Hence why I think Anthropics whole stance of being completely against open anything is actually *extremely* dangerous due to the centralization of power which they completely ignore as a risk factor.
The most discussable thing in this is certainly the idea that labs should fund mathematicians to do expositions.
> One of our principles is that AI labs have a responsibility to provide support, including funding, for the development of human understanding of the AI mathematical output that they release.
Obviously this directionally sounds like the role of mathematicians would be shifting towards interpreting AI results instead of making proofs. I'm not sure who would should really be billed for that. Plus how would you decide who gets the grant?
An interesting thing is that by stating that that the lab that dropped the result provide the funding, this is *directly* proposing an example of taxing AI labs for displacing knowledge workers.
> The use of proprietary internal models by AI labs to do mathematical research risks creating a two-tier system where labs outrun the rest of the field
This is literally the economic bet of the big labs in the broader economy. You use your relative advantage to front run or outcompete.
I'm not sure this argument will work given it is essentially an argument against the thesis the big labs use to justify their valuations.
As an answer to AI companies, this is so bad: instead of trying to find a path to a win-win-ish solution with some trade-offs, this says: sorry, we cannot think of any, so just stop making money, will you? Math needs better crisis managers.
But as an idea, this is even worse: does it mean to stop potential research to cold fusion, cancer and anything as long as it may touch some mathematician's interests, or does it mean math is so hopelessly irrelevant that this cannot be the case... Again, as a crisis manager, this is not how you pose it.
Makes me wonder, were they hired by Sam to sabotage?
For every important match problem solved by AI, without mathematicians we wouldn't know about the existence and importance of the problem.
Famous mathematical conjectures are social constructs, formed by decades of even centuries of attention given to them by members of the math community. Without it, the danger is that future math "progress" will be reduced to generating tables of Lean statements and a probable/unprovable bit generated by AI.
Could just be a sampling bias. Humanity had something like 3000 years to make famous conjectures, whereas AI mathematicians have been around for a month or so. Give them time, I'm sure they'll start formulating highly consequential unsolved problems soon enough.
Those are the two extremes, both are just as bad. Don't excuse shortcomings of the current models with promises of future improvements. But also don't demand literal miracles in 21 business days.
Maybe tiring, but that is the reality. Note that mathematicians are only upset now that "have you tried this on the latest model" works for so many of their problems now, but didn't for the model before that.
> AI mathematicians have been around for a month or so.
LLMs have been around for years, and they're explicitly trained on the entire history of human mathematics (without which they'd be unable to do anything).
In a sense, how do you know whether the problem your AI has just solved is important? A simple proxy is to just check whether humans have thought it's important.
That's also why famous open problems are a good benchmark or proxy: you don't need to convince the rest of the world that the problem your lab's new AI just solved is actually useful or hard.
Or mathematicial progress helps advance society and AI mathematical velocity doesn't offset certain blows to human mathematical velocity yet, so we get to lose progress for the good of an AI company's advertisement.
We don't lose progress by AI solving Navier Stokes. Get a grip on yourselves. I don't think there is a "progress" argument here without tying mathematics to "usefulness", and AI makes mathematics dramatically more useful.
Mathematical progress does (quite obviously, on the whole) help advance humanity, so progress is a good thing. The problem is that defunding mathematicians and handing over control to AI and the companies that create them will cause the subject to stagnate. Sure, for a while we might get progress on existing questions using (perhaps quite novel) combinations of existing techniques, but, so far, given the character of the results we’ve seen, there’s no indication that it will continue indefinitely. Even if it did, what would be the point? Huge textbooks full of work no one can understand or benefit from?
One possible analogy is that humans work to add new points to the space of mathematical knowledge, and AI then fleshes this out to attain the ‘convex hull’ of these points. Essentially, humans ‘invent’ the definitions and pose the questions and AI does the grunt work as well as some creative exploitation of known results and tools to bring down all the low-hanging fruit that follows (important note: what appears to be non-low-hanging fruit to us may in fact be technically low hanging once AI is involved; we saw this for example with the Jacobian conjecture). This seems to be the current situation, and to argue that humans are fully replaced it is necessary to argue that AI is adding points outside the convex hull of human mathematics. A sufficient example would be a first-principles AI proof using alien techniques, and this we haven’t seen so far.
The mathematics-chess comparison is, to put it bluntly, nonsense. I see where it comes from, but, as absolutely anyone with any research experience will tell you, mathematics is orders of magnitude (and this really isn’t strong enough) more open-ended, and doesn’t consist of a game one is seeking to ‘win’. The goal is understanding itself.
I think we need to start looking at another approach, can we create interactive reflex games that upload the knowledge from LLM's or perhaps domain specific ones for formal mathematics, so that mathematicians get a similar level of access to the domain of discourse as the LLM?
I agree. Its so strange to see an institution externalising their specific problems. If they have a problem, they should adapt and fix it amongst themselves.
I saw an interview (in French) of Cédric Villani, speaking on behalf of him and other Fiels medalists, who said that in order to advance mathematics we need three things:
- ideas
- students
- problems
Basically, students to bring original ideas to try and solve existing problems and this generates new ideas and possibly new problems for new students to try and solve with new ideas and so on.
And he continued to say that the issue with LLMs in mathematics, is that he's afraid they could run out of problems, and so students wouldn't bother trying, and this could hurt understanding of mathematics as a whole.
It's totally not my field so I'm not sure what to think of it but it seems important
If youre going to spend 10 million dollars on 10000 agents trying to solve some important maths problem I think it's reasonable to ask for some grants to help digest whatever they came up with.
Or you could hire mathematicians and do it in-house, but I guarantee you grants to PhD students are cheaper than silicon value salaries.
> However, it is now the case that AI can output mathematical arguments in situations without the human who prompted it being able to understand the arguments, verify them, or take responsibility for them.
I read this as "anybody can prompt, few can understand". And "we need more who can understand". If we had more mathematicians (than we have today) all of them piloting advanced models, the pie would grow. The problem is that AI capabilities drain (by disincentivizing) the education pipeline that would get us those mathematicians, and if recent rumbles about what AI is doing to education are to be believed, it does so many years before students even get to grad school.
IMHO, it is not that bad. Not having any human who understands linear algebra after the Butlerian Jihad is a win :-) .
While applying, I looked at the current SoTA, (briefly) read through some of the papers, and realized that I am very far away from understanding them.
Understanding one of these proofs is the work of several months, years or lifetimes depending on whether or not something clicks. It requires a kind of stamina that I quite frankly don't have, but I would like to develop.
If the mathaton's organizers accept my team, I realized that I would spend the next few years working through the result.
So why apply to the Mathaton?
"Many years ago the great British explorer George Mallory, who was to die on Mount Everest, was asked why did he want to climb it. He said, 'Because it is there.'
Well, [theoretical math] is there, and we're going to climb it, and [topology] and [number theory] are there, and new hopes for knowledge and peace are there. And, therefore, as we set sail we ask God's blessing on the [~~most hazardous and dangerous and greatest adventure~~] on which [we have] ever embarked."
More seriously, I applied because I was hoping to get access to the models without the veil. I don't think people realize just how big the gap is between what exists behind the scenes at these entities, and what we get out here.
And it's frustrating. Because I think it's within the rights of frontier labs to decide whether or not to sell access to a product, but the labs aren't just doing that. They're trying to thumb the scale to make sure that none of us ever get access to these models at peak performance. Ever.
And I think humanity is worse off for that. I am worse off for that.
I have studied the shape and structure of historical technological revolutions (and I've written about it), and usually the world doesn't realize how big of a big deal the big deal is because the big deal is often flawed, broken, and under-delivers. In the short term.
In the long term...? The world changed in the past few months. I think mathematics is one small part of that.
For most of human history, higher mathematics would have been inaccessible to me, and other outsiders, no matter how well heeled. Mathematics is, or rather was, a living discipline that existed piecemeal in a handful of minds across the world. These people's time was finite and valuable. To just meet them, you'd have to jump through hoops, and spend years proving yourself.
There is no price for an hour of tutoring from Terence Tao. But now, with AI? You can have an entity with the capabilities of Terry Tao help you understand the subtleties of math.
AI has changed what mathematics is. And every prominent mathematician seems to know it. They feel like mathematics has been devalued, and in some ways it has. Mathematics has gone from being a living discipline kept alive by a chosen few to a wellspring everyone can sip from. For the first time in human existence, learning and accessing higher mathematics doesn't involve jumping through hoops and knowing the right people. You can just ask.
I can just ask.
Except I can't. Because that capability is being gate kept. And I want to know. I want to climb the mountain.
"There is no royal road to Geometry" - Euclid. Mathematical knowledge isn't from tutoring but doing. And no one is against the AI helping explain the tricky parts to help you practice. Rather it's about dumping a giant bunch of low quality text with some Lean claiming a big result is done.
There are subtleties to mathematics that aren't easy to understand from the written page alone. It's why it's a living medium.
For example, as we're talking about LLMs... why not, there are ways to reason about vector spaces that weren't intuitive for me to understand. It's something that required talking things out with a friend who is a practising mathematician (albeit in training).
I am not smart enough to reconstruct all of mathematics on my own from scratches on paper alone. That back and forth is necessary. And it's something that you couldn't have "bought" for cutting edge math at any price a few months before this point in time. Because it exists in the minds of people and it needs lots of back and forths with those people.
It's why LLM proofs can be slop on paper. A proof that no one can check or understand is not but scratches on paper. BUT LLMs are also the solution to the problem they create. The machines that can generate proofs are also machines that can help us understand them.
The living medium can now be represented and scaled inside of a machine. I can now sit down at an airport and have that discussion. I think that's transformative for our species.
Just wait until BCI and/or fast Pavlovian conditioning force fed by agents into human learners.
I've been vibe coding my own SRS software that is vastly superior to my learning style than Anki, and I know I'm just scratching the surface of accelerated learning. Who knows where this goes.
MRI and glucose injections with agent-tutor steering and millisecond feedback to learning?
We might be able to Matrix "I Know Kung-Fu" things into brains one day.
Can you, I, or any mathematician who isn't well connected (let's say someone who is a young Maryam Mirzakhani or just someone who is in grad school) learn from the system that produced the solution to the unit distance problem? https://cdn.openai.com/pdf/74c24085-19b0-4534-9c90-465b8e29a...
You will notice that it says on the first page,
"first mathematically generated in one shot by an internal model at OpenAI"
And I want to talk to models of similar aptitude and capability to help me understand nuances of the proof. Mathematicians will happy to pay for this. I've heard that people and non-profits are putting together $$$ for this to get access to these systems so that they can all interrogate them.
But the issue is that we can't. And I'm using the royal we here.
The paper says that the labs shouldn't release proofs from models that mathematicians can't interrogate. It's very clear that the models we get as users aren't the models used to produce the breakthroughs. And as LLMs display emergent capabilities, it's uncertain whether or not the model actually understands what it's explaining.
Because if I don't understand it. Professional mathematicians who are subject experts don't understand it. Then how do we know the model does? How do we know that it's correctly representing the proof produced by a more capable model? It's not logical to take any random model at its word, unless we can verify. Or, if it's the same model that produced the proof.
And that's what the mathematicians want. Access to the actual models.
OpenAI should be allowed to produce whatever it wants but it just can't claim that it has actually solved without the due process like peer review. If for example OpenAI solves a new conjecture, OpenAI should be free to publish it in their blog or arxiv in whatever way they desire. It can be slop, it can be non-slop. No one should police it.
Mathematicians are free to use it or discard it. They shouldn't externalise their concerns and restrict labs.
Mathematics is seen today as the noblest and most aristocratic of professions. Turns out, AI disrupts it because access to capital/compute now decides the results. Mathematicians don't like this corruption - understandable.
Its like guild of accountants opposing the calculator and require a responsible release. haha
> we ask them to stop testing advanced mathematical problems on proprietary models.
Maybe I'm alone on this, but for some reason these sorts of requests strike me as akin to gatekeeping how someone should breathe air. It's math... the numbers and symbols are just out there in the platonic realm available for anyone to do as they like with them. It's patently absurd to request other people to stop.
Ensuring credit where credit is due? That's fine. If your model incorporates the efforts of many others, then it's reasonable to request acknowledgement of everyone who contributed (even indirectly). But that's not what the request states — presumably their ask subsumes any advanced ML model, including those that weren't trained on a giant corpus of text.
> akin to gatekeeping how someone should breathe air.
The law is also a sequence of letters available to anyone. Yet you cannot practice without passing the bar exam. And that is not the only example; many professions can only be practiced by licensed individuals: medical doctors, journalists, electricians...
To quote Noam Brown, "Our main focus is shipping great models so everyone can use them to make discoveries of their own" [1]. Spending 15 million dollars of compute with an internal model to blitz and scoop a resolution of Navier-Stokes that someone is already on track to resolve (or, from my understanding of why OpenAI did this, had already resolved, as staff at OpenAI stated their attempt was prompted by rumors of a resolution by Anthropic) is in complete opposition to this stated claim. That act is what this portion of the recommendation is in response to: OpenAI perpetually holding out internal models and tooling, using them to solve important problems in math, and thus themselves holding a monopoly on certain aspects of mathematics. Why is it absurd to ask people who are in a position to do an obviously damaging act to not do so, especially when those same people were the ones who asked for your advice in the first place and claimed to not want to do said act previously?
[1] https://x.com/polynoamial/status/2093451221273387477
>was prompted by rumors of a resolution by Anthropic
reminds an old sci-fi story where top engineers were shown a video of a genius inventor who invented anti-gravity and unfortunately died while testing his apparatus which was clearly anti-gravitating in the video. Thus believing that it is a solved problem, just need to rediscover the lost solution, the engineers quickly developed anti-gravity. The original video happened to be a fake specially created for that purpose.
Because it’s not a damaging act, to those who aren’t gatekeepers.
> akin to gatekeeping how someone should breathe air.
Yes, and professions that do that - that vocally insist there is an essential human element to the craft - will likely fare better than the ones that say "whatever, code is code" or "whatever, math is math".
This is smart. We might not like it, but SWEs are actually the dumb ones with their utilitarian attitudes to AI. We're digging our own grave. Meanwhile, professions such as writers or musicians are positioning themselves better by shunning "artists" that simply pull the lever of a gen AI system.
The problem with proprietary models is that you don't get to poke inside and see what it is doing and how it arrives at its answer, which is precisely what mathematicians do. Mathematical understanding derives less from any particular result than the insights and methods that pave the road to results. Whenever a theorem is proved, researchers seek to unpack the proof and get inside the author's mind to learn their ways of thinking.
LLM generated results might benefit mathematical understanding if people can inspect their intermediate reasoning traces to discover erroneous human biases or patterns that they might have previously overlooked. Otherwise, the results might as well be produced by oracles.
The proof itself can be studied, either with AI or not.
"Intermediate reasoning traces" might be interesting but they are not by any means requited.
This letter has nothing to do with reasoning traces anyway: they just don't want math research to be front-run by internal models, that's all.
> The problem with proprietary models is that you don't get to poke inside and see what it is doing and how it arrives at its answer, which is precisely what mathematicians do
Uh, a hypothetical fully open source model would have the exact same problem, because LLMs rely on emergent phenomena and no one understands why they work.
>Otherwise, the results might as well be produced by oracles.
no. The math result is a result only when it includes proof. The proof is the value here. The way somebody came to it isn't really important - we don't know how Newton came to his results, whether it was apple or pear, and it isn't really important. Or how Einstein was walking the city streets looking at the tower watches - it is just historic curiosity having no real value for science.
That has been one of the greatest thing about math departments - smooth talkers were always clearly visible as smooth talkers. You're either producing proofs, or you're anything but a mathematician.
I feel for mathematicians. They have similar situation like we have in programming. Well, we all just have to evolve and adjust (in particular reign in our pride as just in a few years - i think once LLMs start hitting 100T+ - we may loose our "top of God's creation" position). Any attempts at gatekeeping, ludditing, organizing in quasi observational/advisory boards really intended to protect their tenures, etc. ... - well, you just can't stop the wave.
It all reminds how Catholic Church insisted on responsible release of the Bible in German. The Church even unleashed the devastating 30 Years War trying to protect its monopoly on religion including the right to sell indulgences, etc.
>AI labs should provide significant support, including funding
And now all those "responsible math" and advisory boards would like to preserve their monopoly on math and would like to sell the indulgences to the AI labs. As usually it is all about money and power, not about science. As a Math PhD dropout myself i feel a bit of a shame and disappointment for that undignified scramble by the mathematics establishment. Being smart they should have led the way and show an example to the rest of humanity ...
> requests strike me as akin to gatekeeping
This is exactly what I was thinking as I read it, along with the bit that AI labs should support human understanding. The whole thing smells as they're trying to place this burden on labs that are just offering a tokens service; them publishing about particular topics is essentially a side quest in the first place IMO. If a community wants to create Math labs dedicated to understanding AI discoveries in the field, then they're free. If they want to petition AI labs for financial support, they're also free. But this wording where they're trying to dictate what AI labs should do (outside of their primary business) just smells.
But on one hand there are labs that pushes tens of millions $ to mine for publicity, and on one hand mostly underfunded researchers trying to improve general understanding. Big ai may seriously harm math and when Pr value diminishes down nobody is there to keep pushing.
“researchers trying to improve general understanding”
Pretty sure AI will do better for that.. mathematicians need to be centaurs like the rest of us and stop rhetoric that is going to make existing math centaurs feel like they might get math-cancelled
My point is that there is a real difference between massive ai company budgets and research mathematicians driven it, it is not an argument against use of ai which seems inevitable.
>mathematicians need to be centaurs like the rest of us and stop rhetoric that is going to make existing math centaurs feel like they might get math-cancelled
Centaurs?
I don't see why mathematicians should be protected from AI anymore than any other profession. It's either everybody or nobody, not fair on the face of it otherwise.
TFA isn't asking for mathematicians to be protected from AI. It's asking AI labs to hold themselves to the standards of the mathematical community:
- releasing papers using the normal process to allow peer review
- giving talks etc to disseminate knowledge so humans understand the result
- writing papers in a way (standard terminology etc) that allows mathematicians to digest the result (some AI math papers comprise a huge verbose load of non-standard terminology and waffle and then a massive lean proof. This is very hard for humans to actually understand, and means it's hard for others to take the work forward.)
- giving appropriate credit to results that are used to derive the work
It includes some specific recommendations for situations where the person prompting the model is not in a position to understand the output, and frankly these are really welcome given situations like the recent case at Anthropic where a non-mathematician at Anthropic prompted claude to make a significant improvement to the bounds of a problem related to the Riemann Zeta function[1] which led to widespread misreporting and claims (not by Anthropic themselves notably) that the Riemann hypothesis itself had been proved, which is emphatically not the case.
Research mathematics is fundamentally a collaborative activity and the way in which some of these results are released is done to maximise PR but means a ton of the mathematical value is left on the table.
[1] https://www.anthropic.com/research/riemann-zeta. As I understand it, the Riemann Hypothesis says that all non-trivial zeroes of the zeta function lie on a line called the critical line. Two centuries of previous work had established that at least something like 40.9% of the zeroes lie on the line and noone has ever found a non-trivial zero that does not lie on that line. Claude (with prompting from a non-mathematician to "try harder" etc) improved this bound massively to 67%. Now a lot of people said things like "OK so all we've got to do is to improve that to 100% and we've proved the RH", which is definitely not true unfortunately, because you can say that in the limit the proportion of the zeroes on the line is 100% and still have infinitely many which are not.
From the opening paragraph:
we do not endorse this practice, and we ask them to stop testing advanced mathematical problems on proprietary models
From point 2 under section "1. Background":
AI labs should provide significant support, including funding, to help develop this understanding
Point 3 under the same section:
The development of human understanding must remain organic and community led. It should not be directed by AI labs, even when the labs have produced the results.
If, as you say, they are "asking AI labs to hold themselves to the standards of the mathematical community", I must conclude that the mathematical community
1. Ideally wants a monopoly on mathematics research.
2. Demands money from those who dare violate their ideal monopoly.
3. Insists that the ideal monopoly remain in charge.
Math is a tool. It's what people do with the tools after they are invented that matters - and with mathematical tools sometimes that wait is hundreds of years.
The mathematics community has found that the tools have value long after being invented if the inventor leaves his tools in a specific format. The community is doing its best to preserve that - not because they want less toolmakers, but rather because they want the tools to be useful when they become needed.
Maintaining the existing standards is in fact a form of protection from AI disruption. They are asking the AI companies to follow their norms, instead of them having to conform to new norms created by AI. They don’t want to have to change the way they do things, understandably!, and are asking the companies to accommodate their way of life.
Those seem reasonable apart from this one:
> - releasing papers using the normal process to allow peer review
I'm not an academic but I've heard enough stories about how this can very much act as gatekeeping that I don't think it is a good request.
Mathematician was never really a "profession" like the others. It doesn't pay well and is largely confined to academia. If you're really good and want to get paid, you don't do the kinds of problems AI have been taking a crack at. You go to a quant firm or some tech company where this kinda math actually matters once in a blue moon
You're not alone, even among mathematicians.
You’re not alone at all, gatekeeping is exactly what this is and that’s clear from the text almost immediately.
Humans have been trying to outlaw thinking for some time now.
What (some) mathematicians are asking for, whether one agrees with it or not, is exactly the opposite of 'outlawing thinking'.
Note that they're saying "proprietary models". Probably open models will reach this level in a year or so and then this discussion will be moot.
> gatekeeping how someone should breathe air
Air is there for all to breathe. It's a more-or-less fungible, free resource for all to use and the consumption of it is a basic requirement for life.
The AI companies, in contrast are using vast financial, human and compute resources to train and operate specialised models that are not available to the public. The advisory group's job is to give non-binding advice on how they can use this privately owned technology in a responsible manner that avoids doing unnecessary harm to the mathematical community. To call that gatekeeping misses the point entirely.
The AI models are most definitely available to the public with a maximum of what, a 3 month lead while they do testing?
The very first paragraph of the article says (emphasis mine):
> At present, some frontier AI labs are testing advanced mathematical problems on proprietary models that remain inaccessible to the broader scientific community. Our recommendations are formulated with this practical context in mind.
The Navier Stokes proof came from an internal model that AFAIK still has not been released even in a limited way to scientists, let alone to the general public. Publicly available models are not what these mathematicians are talking about.
Air is not impossible to disrupt in quality as side effect of industrial hubris.
> gatekeeping how someone should breathe air. It's math... the numbers and symbols are just out there in the platonic realm available for anyone to do as they like with them. It's patently absurd to request other people to stop.
The chemical compounds of all kinds are also out there, available for anyone to do as they like with them. Say, mixing ammonium chlorate with peroxide, why not? Or potassium permanganate with powdered aluminum. It's patently absurd to request other people to stop.
It's illegal to make weapons out of chemicals because the government has a monopoly on violence.
Mix together whatever you want, and then you can fight the feds when they show up to enforce their monopoly.
> The chemical compounds of all kinds are also out there,
1. Lol but they're literally not
2. A sample of chemical compound and a piece of math don't share literally any ontological qualities - you might as well have tried to make a comparison between math and nude pictures of the President
They literally are though.
> you might as well have tried to make a comparison between math and nude pictures of the President.
The latter is but a very large number, interpreted in a particular way. Have you heard of "illegal numbers"? Yeah, apparently they exist. That damn state, treading on literally everything that humans might do as if it's any of its business.
Link for curious people
https://en.wikipedia.org/wiki/Illegal_number
Mostly pretty straight forward. The idea that proofs be desloppified, attribute existing literature properly, and published somewhere expediently where it can be commented on, with artifacts for verification, is all very uncontroversial stuff.
I think the spicy take is definitely this stance that longstanding mathematical problems shouldn't be used as benchmarks for (specifically proprietary) models. Stated right at the very top.
The justification is pretty clear.
> The use of proprietary internal models by AI labs to do mathematical research risks creating a two-tier system where labs outrun the rest of the field, effectively alienating the mathematical community from its own discipline.
It is all fun and games (for non mathematicians) when mathematicians can't compete with AI labs but I think the more dangerous direction is when this starts being true for the rest of everything. For example cybersecurity or whatnot. Hence why I think Anthropics whole stance of being completely against open anything is actually *extremely* dangerous due to the centralization of power which they completely ignore as a risk factor.
The most discussable thing in this is certainly the idea that labs should fund mathematicians to do expositions.
> One of our principles is that AI labs have a responsibility to provide support, including funding, for the development of human understanding of the AI mathematical output that they release.
Obviously this directionally sounds like the role of mathematicians would be shifting towards interpreting AI results instead of making proofs. I'm not sure who would should really be billed for that. Plus how would you decide who gets the grant?
An interesting thing is that by stating that that the lab that dropped the result provide the funding, this is *directly* proposing an example of taxing AI labs for displacing knowledge workers.
> The use of proprietary internal models by AI labs to do mathematical research risks creating a two-tier system where labs outrun the rest of the field
This is literally the economic bet of the big labs in the broader economy. You use your relative advantage to front run or outcompete.
I'm not sure this argument will work given it is essentially an argument against the thesis the big labs use to justify their valuations.
As an answer to AI companies, this is so bad: instead of trying to find a path to a win-win-ish solution with some trade-offs, this says: sorry, we cannot think of any, so just stop making money, will you? Math needs better crisis managers.
But as an idea, this is even worse: does it mean to stop potential research to cold fusion, cancer and anything as long as it may touch some mathematician's interests, or does it mean math is so hopelessly irrelevant that this cannot be the case... Again, as a crisis manager, this is not how you pose it.
Makes me wonder, were they hired by Sam to sabotage?
For every important match problem solved by AI, without mathematicians we wouldn't know about the existence and importance of the problem.
Famous mathematical conjectures are social constructs, formed by decades of even centuries of attention given to them by members of the math community. Without it, the danger is that future math "progress" will be reduced to generating tables of Lean statements and a probable/unprovable bit generated by AI.
Could just be a sampling bias. Humanity had something like 3000 years to make famous conjectures, whereas AI mathematicians have been around for a month or so. Give them time, I'm sure they'll start formulating highly consequential unsolved problems soon enough.
Somewhat tiring that as alway any criticism is reduced to "but have you tried this on the latest model".
Those are the two extremes, both are just as bad. Don't excuse shortcomings of the current models with promises of future improvements. But also don't demand literal miracles in 21 business days.
Maybe tiring, but that is the reality. Note that mathematicians are only upset now that "have you tried this on the latest model" works for so many of their problems now, but didn't for the model before that.
> AI mathematicians have been around for a month or so.
LLMs have been around for years, and they're explicitly trained on the entire history of human mathematics (without which they'd be unable to do anything).
Yes, and they've done absolutely fuck all with this knowledge until very, very, very recently.
Not just sampling bias, but also human bias.
In a sense, how do you know whether the problem your AI has just solved is important? A simple proxy is to just check whether humans have thought it's important.
That's also why famous open problems are a good benchmark or proxy: you don't need to convince the rest of the world that the problem your lab's new AI just solved is actually useful or hard.
Either mathematical progress helps advance society, in which case progress is a good thing.
Or mathematics is more like a hobby, and while ai may spoil their fun, they need to move on like chess and go players.
Or mathematicial progress helps advance society and AI mathematical velocity doesn't offset certain blows to human mathematical velocity yet, so we get to lose progress for the good of an AI company's advertisement.
We don't lose progress by AI solving Navier Stokes. Get a grip on yourselves. I don't think there is a "progress" argument here without tying mathematics to "usefulness", and AI makes mathematics dramatically more useful.
That’s a false dichotomy.
Mathematical progress does (quite obviously, on the whole) help advance humanity, so progress is a good thing. The problem is that defunding mathematicians and handing over control to AI and the companies that create them will cause the subject to stagnate. Sure, for a while we might get progress on existing questions using (perhaps quite novel) combinations of existing techniques, but, so far, given the character of the results we’ve seen, there’s no indication that it will continue indefinitely. Even if it did, what would be the point? Huge textbooks full of work no one can understand or benefit from?
One possible analogy is that humans work to add new points to the space of mathematical knowledge, and AI then fleshes this out to attain the ‘convex hull’ of these points. Essentially, humans ‘invent’ the definitions and pose the questions and AI does the grunt work as well as some creative exploitation of known results and tools to bring down all the low-hanging fruit that follows (important note: what appears to be non-low-hanging fruit to us may in fact be technically low hanging once AI is involved; we saw this for example with the Jacobian conjecture). This seems to be the current situation, and to argue that humans are fully replaced it is necessary to argue that AI is adding points outside the convex hull of human mathematics. A sufficient example would be a first-principles AI proof using alien techniques, and this we haven’t seen so far.
The mathematics-chess comparison is, to put it bluntly, nonsense. I see where it comes from, but, as absolutely anyone with any research experience will tell you, mathematics is orders of magnitude (and this really isn’t strong enough) more open-ended, and doesn’t consist of a game one is seeking to ‘win’. The goal is understanding itself.
I think we need to start looking at another approach, can we create interactive reflex games that upload the knowledge from LLM's or perhaps domain specific ones for formal mathematics, so that mathematicians get a similar level of access to the domain of discourse as the LLM?
I agree. Its so strange to see an institution externalising their specific problems. If they have a problem, they should adapt and fix it amongst themselves.
Whatabout mathematical progress that ruins humanity even more ? e.g a much more addicted algorithm than tiktok/facebook reel.
I saw an interview (in French) of Cédric Villani, speaking on behalf of him and other Fiels medalists, who said that in order to advance mathematics we need three things:
- ideas
- students
- problems
Basically, students to bring original ideas to try and solve existing problems and this generates new ideas and possibly new problems for new students to try and solve with new ideas and so on.
And he continued to say that the issue with LLMs in mathematics, is that he's afraid they could run out of problems, and so students wouldn't bother trying, and this could hurt understanding of mathematics as a whole.
It's totally not my field so I'm not sure what to think of it but it seems important
This paper wants AI companies to pay human mathematicians to understand AI-generated stuff. That's an unusual ask.
If youre going to spend 10 million dollars on 10000 agents trying to solve some important maths problem I think it's reasonable to ask for some grants to help digest whatever they came up with. Or you could hire mathematicians and do it in-house, but I guarantee you grants to PhD students are cheaper than silicon value salaries.
Why pay for grants when you can just spend the money on compute more efficiently?
> However, it is now the case that AI can output mathematical arguments in situations without the human who prompted it being able to understand the arguments, verify them, or take responsibility for them.
I read this as "anybody can prompt, few can understand". And "we need more who can understand". If we had more mathematicians (than we have today) all of them piloting advanced models, the pie would grow. The problem is that AI capabilities drain (by disincentivizing) the education pipeline that would get us those mathematicians, and if recent rumbles about what AI is doing to education are to be believed, it does so many years before students even get to grad school.
IMHO, it is not that bad. Not having any human who understands linear algebra after the Butlerian Jihad is a win :-) .
I applied to the caltech mathaton.
While applying, I looked at the current SoTA, (briefly) read through some of the papers, and realized that I am very far away from understanding them.
Understanding one of these proofs is the work of several months, years or lifetimes depending on whether or not something clicks. It requires a kind of stamina that I quite frankly don't have, but I would like to develop.
If the mathaton's organizers accept my team, I realized that I would spend the next few years working through the result.
So why apply to the Mathaton?
"Many years ago the great British explorer George Mallory, who was to die on Mount Everest, was asked why did he want to climb it. He said, 'Because it is there.'
Well, [theoretical math] is there, and we're going to climb it, and [topology] and [number theory] are there, and new hopes for knowledge and peace are there. And, therefore, as we set sail we ask God's blessing on the [~~most hazardous and dangerous and greatest adventure~~] on which [we have] ever embarked."
More seriously, I applied because I was hoping to get access to the models without the veil. I don't think people realize just how big the gap is between what exists behind the scenes at these entities, and what we get out here.
And it's frustrating. Because I think it's within the rights of frontier labs to decide whether or not to sell access to a product, but the labs aren't just doing that. They're trying to thumb the scale to make sure that none of us ever get access to these models at peak performance. Ever.
And I think humanity is worse off for that. I am worse off for that.
I have studied the shape and structure of historical technological revolutions (and I've written about it), and usually the world doesn't realize how big of a big deal the big deal is because the big deal is often flawed, broken, and under-delivers. In the short term.
In the long term...? The world changed in the past few months. I think mathematics is one small part of that.
For most of human history, higher mathematics would have been inaccessible to me, and other outsiders, no matter how well heeled. Mathematics is, or rather was, a living discipline that existed piecemeal in a handful of minds across the world. These people's time was finite and valuable. To just meet them, you'd have to jump through hoops, and spend years proving yourself.
There is no price for an hour of tutoring from Terence Tao. But now, with AI? You can have an entity with the capabilities of Terry Tao help you understand the subtleties of math.
AI has changed what mathematics is. And every prominent mathematician seems to know it. They feel like mathematics has been devalued, and in some ways it has. Mathematics has gone from being a living discipline kept alive by a chosen few to a wellspring everyone can sip from. For the first time in human existence, learning and accessing higher mathematics doesn't involve jumping through hoops and knowing the right people. You can just ask.
I can just ask.
Except I can't. Because that capability is being gate kept. And I want to know. I want to climb the mountain.
"There is no royal road to Geometry" - Euclid. Mathematical knowledge isn't from tutoring but doing. And no one is against the AI helping explain the tricky parts to help you practice. Rather it's about dumping a giant bunch of low quality text with some Lean claiming a big result is done.
Yes, it's why I love math. You can't buy fluency.
There are subtleties to mathematics that aren't easy to understand from the written page alone. It's why it's a living medium.
For example, as we're talking about LLMs... why not, there are ways to reason about vector spaces that weren't intuitive for me to understand. It's something that required talking things out with a friend who is a practising mathematician (albeit in training).
I am not smart enough to reconstruct all of mathematics on my own from scratches on paper alone. That back and forth is necessary. And it's something that you couldn't have "bought" for cutting edge math at any price a few months before this point in time. Because it exists in the minds of people and it needs lots of back and forths with those people.
It's why LLM proofs can be slop on paper. A proof that no one can check or understand is not but scratches on paper. BUT LLMs are also the solution to the problem they create. The machines that can generate proofs are also machines that can help us understand them.
The living medium can now be represented and scaled inside of a machine. I can now sit down at an airport and have that discussion. I think that's transformative for our species.
> You can't buy fluency.
Just wait until BCI and/or fast Pavlovian conditioning force fed by agents into human learners.
I've been vibe coding my own SRS software that is vastly superior to my learning style than Anki, and I know I'm just scratching the surface of accelerated learning. Who knows where this goes.
MRI and glucose injections with agent-tutor steering and millisecond feedback to learning?
We might be able to Matrix "I Know Kung-Fu" things into brains one day.
You might need to add in chemically altered states of consciousness and magnetic stimulation of the right areas, but yes.
> Except I can't. Because that capability is being gate kept.
In what way are these "gatekeepers" stopping you asking an LLM questions about maths?
I would like to ask you a question.
Can you, I, or any mathematician who isn't well connected (let's say someone who is a young Maryam Mirzakhani or just someone who is in grad school) learn from the system that produced the solution to the unit distance problem? https://cdn.openai.com/pdf/74c24085-19b0-4534-9c90-465b8e29a...
You will notice that it says on the first page,
Mathematicians want to talk to the exact model variant whose summarized chain of thought is, https://cdn.openai.com/pdf/1625eff6-5ac1-40d8-b1db-5d5cf925d...And I want to talk to models of similar aptitude and capability to help me understand nuances of the proof. Mathematicians will happy to pay for this. I've heard that people and non-profits are putting together $$$ for this to get access to these systems so that they can all interrogate them.
But the issue is that we can't. And I'm using the royal we here.
The paper says that the labs shouldn't release proofs from models that mathematicians can't interrogate. It's very clear that the models we get as users aren't the models used to produce the breakthroughs. And as LLMs display emergent capabilities, it's uncertain whether or not the model actually understands what it's explaining.
Because if I don't understand it. Professional mathematicians who are subject experts don't understand it. Then how do we know the model does? How do we know that it's correctly representing the proof produced by a more capable model? It's not logical to take any random model at its word, unless we can verify. Or, if it's the same model that produced the proof.
And that's what the mathematicians want. Access to the actual models.
Honestly what is "AI generated math"?
Everything is WFFs all the way down
I disagree with this.
OpenAI should be allowed to produce whatever it wants but it just can't claim that it has actually solved without the due process like peer review. If for example OpenAI solves a new conjecture, OpenAI should be free to publish it in their blog or arxiv in whatever way they desire. It can be slop, it can be non-slop. No one should police it.
Mathematicians are free to use it or discard it. They shouldn't externalise their concerns and restrict labs.
Mathematics is seen today as the noblest and most aristocratic of professions. Turns out, AI disrupts it because access to capital/compute now decides the results. Mathematicians don't like this corruption - understandable.
Its like guild of accountants opposing the calculator and require a responsible release. haha
There’s a reason why many of your posts are downvoted.
Luddites are life-deniers and engage in extreme amounts of slave morality. Reject them with extreme prejudice in every context.