I think it's a useful analogy to compare OpenAI to a human collaborator. These researchers willingly collaborated with an OpenAI model, giving it ideas, and OpenAI provided useful replies. Then, OpenAI goes ahead and publishes work along the lines of this collaboration, without attributing the researchers. If OpenAI was in fact a human researcher, this would be highly unethical.
Now, OpenAI is claiming that the model it used to generate the result was not trained on these collaborative communications with the researcher. This is a technical argument that is impossible to verify as an OpenAI outsider, and probably difficult to verify even for internal OpenAI employees. Provenance is hard to track - you would hope OpenAI has very good tools for this, but a full data trail of all inputs is difficult to trace through.
Another interesting thing to consider is if instead of OpenAI doing this, it was another research mathematician A using an OpenAI model just like the internal group at OpenAI did to publish these results. What if the model A used was trained with unpublished communications with other researchers B who were working on the same problem? Should researcher A technically include B as coauthors? How could they do this when they do not know the communications B had with OpenAI? In this scenario OpenAI, as a middle man, has laundered information from B to A, stripping out attribution. A scooped B without even knowing it!
First, OpenAI is not claiming that the model wasn't trained on those sessions. What they've said is “We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem.” and “We did not use their prompts or proofs to prompt our models or direct our agents.” and “While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”
They also said “Our effort began on September 1st after hearing a rumor which we later realized was related to Levent Alpöge … and Tristan Buckmaster….” They say the rumor was that two Millennium Prize problems had been resolved, and that this prompted them to launch "an effort to evaluate it on all open Millennium Prize problems and a few other high-impact problems."
It's not obvious to me that's an unethical thing to do, if it happened as they described.
> It's not obvious to me that's an unethical thing to do
In terms of work in mathematics, something I personally would not do based on ethical grounds would be to hear a rumor that some researchers are taking a certain approach and may be nearing a solution, use a model that was possibly contaminated with intimate knowledge about that approach (though later they investigated and think it wasn't), and then commit millions to tens of millions of dollars and untold amounts of hardware to try to beat them to it. If I had done this, I also wouldn't have pestered the researchers on a Sunday night to meet immediately so we could negotiate a nice way of presenting the actions I had decided to take.
Even if you don't think it was unethical, it was never going to be received well in the community that was especially going to care about this work, and who are very much peers to many of the people working on this solution, so it was at the least an enormous (and well-deserved) own-goal that their unveiling of their solution to NS went like this.
But by OpenAI's telling they heard a rumor that the problem had already been solved. So they reached out to the other researchers as an attempt to share the credit, and in fact have at least one of them be the lead author (which is when they found out the AI had solved a broader problem than the researchers.) Seems pretty ethically palatable.
I suspect the main reason the community is not receiving it well is largely the same reason many developers are not receiving coding agents well.
Even if you judge OpenAI solely on their public communications it still sounds really bad.
That they heard a rumour that a major open problem had been solved, so they decided to try and scoop the other mathematicians while they were writing up their preprint is extremely unsporting.
Then they decided to exclude an author because of his employer, even though he had used their own products to write the proof!
They haven't necessarily breached any formal ethical rules but their behaviour will lead to them and their products being shut out from the mathematical community.
> I suspect the main reason the community is not receiving it well is largely the same reason many developers are not receiving coding agents well.
Because the training data is millions of hours human efforts being distilled into a cascading hierarchy of enrichment by interested parties without providing attribution or compensation?
Well, the investment dollars are spent on the customers for the most part, though also on salaries and equipment. But the lions share of the value is going to the shareholders (eg employees and investors)... and they have liquidated and will continue to liquidate a disproportionate value to what they have spent on us. By some estimations at least. It's very possible $1 into this machine to feed your queries is worth $10+ to a shareholder based on whatever new valuation they get. So I'd say there is a hierarchy of enrichment.
many teachers also taught many students over the course of history, and very few would eventually pay any compensation or even attribute their financial (or career) outcomes to the teachers.
Huh? In your example these many teachers were paid for teaching these students and were able to make a living off of teaching without the students compensating or attributing their financial (or career) outcomes to the teachers while now we have a system where we are expected to pay a monthly amount to a corporation that has inhaled all human knowledge without any financial compensation to the people who created, managed or maintained this knowledge. The effective difference being that our knowledge, which used to be a means of income, has now become a subscription cost.
as a developer that had a brief career in academia, i don't think your last comment is right at all. 99.9% of what i work on as a webdev, even if it's challenging and unique at the margins, is not really novel. concerns about job security aside, i don't really think of an agent as stealing my ideas because it's good at writing CRUD APIs.
collaborating with ChatGPT on a novel solution to an unsolved problem, getting 90% of the way there, and then being "scooped" by your AI collaborator (or rather by the company behind it) is a totally different situation. were i in the same situation as these researchers, it would be extremely hard to take OpenAPI's explanation + denial of plagiarism seriously
But that is exactly what I'm implying is the core reason, whether people realize it or not.
I totally agree that the vast majority of software dev is not novel. I have even made several comments to that effect. The same can be said for a lot of creative work as well. Yet many, many devs and creators are very unhappy with AI, and a lot of their complaints are variations on accusations of plagiarism.
And note, I am not saying it is wrong, it is completely understandable, but we need to be clear about where this turmoil is coming from.
If I were in the same situation as these researchers, I would publish all pertinent research work and chats so that the rest of the world can see how close the model's work is to my own. It's been scooped anyway, so there is no reason to keep it private.
Maybe not money directly, but pretty sure it's about economic disruption. These models directly undercut the value of one's skills and labor, regardless of whether this value is measured in hard cash or abstract self-worth.
I am working on two applications using ChatGPT and Claude.
I have no illusions these people won't steal/copy whatever you want to call it, "train their models".
Yes, I keep unticking the boxes that allow it, that they so kindly tick for me.
But what happened to these math researchers is something else and I am not sure it's about the money for them. You don't do math research to get rich, but to get acknowledged by your peers. Yes, we live in a capitalist world so obviously you need money to feed yourself. but for some people, that is secondary.
OpenAI stole their thunder, and that's just fucked up.
It's not equivalent to cranking out a CRUD app for profit.
Being displaced from a vocation that they either have dedicated their professional lives getting good at, or was their livelihood, or likely, both.
I think all other complaints from all other people in all their myriad variations stem from this core reason. Even if people don't realize it themselves.
Like, if these models had trained on the entirety of human knowledge and art, and then turned out to be absolutely useless, I would bet nobody would waste a second's thought on them.
I'm not sure that's true. Like if they produced nothing but the worst of the slop they're currently producing, a lot of people would still be bothered by that just because of the sheer volume of such slop that can now be produced.
> If I had done this, I also wouldn't have pestered the researchers on a Sunday night to meet immediately so we could negotiate a nice way of presenting the actions I had decided to take.
From what I can tell, both OpenAI and the researchers agree on this meeting happening, except both sides clearly have very different interpretations of what happened and why.
Hearing that something is solvable is already a hint. I don’t think leveraging this knowledge is ethical. They could go after a different problem but didn’t.
The flip side is that the Anthropic researcher is clearly pushing the case against Open AI and one might have reason to suspect their motivation and version of events for the same reasons.
> the Anthropic researcher is clearly pushing the case against Open AI
Things like the nytimes interview are with Buckmaster, who works at NYU, not Alpöge. I saw a couple of tweets from him over the last week. Any chance of clarifying what makes you think he's "clearly pushing the case"?
> The flip side is that the Anthropic researcher is clearly pushing the case against Open AI and one might have reason to suspect their motivation and version of events for the same reasons.
I haven't seen any evidence of this. Much of the anger is coming from the unaffiliated researcher. levent (the anthropic employee) has mostly constrained his comments to basically "I would have been happy to collaborate w/ folks from OAI"
I think people are too reserved in their unwillingness to operationalize ambiguity. Ambiguity is constantly being thrown in our face, with internal audits and other laughable attestations of virtue that amount to a pantomime of transparency / good faith.
Why should I care if a company claims they find no evidence of wrongdoing? Is that the threshold for privacy/trust? “We don’t care if it appears that we’ve been dishonest unless there’s hard proof.” They can simply design proof keeping to terminate at the places their dishonesty is implemented.
For me, when there is a clear motive to be dishonest, a corporation should be assumed to be dishonest unless there are robust transparency measures and a regulatory environment shown to be providing a cost to dishonesty. Without it, all you do is burden yourself while the powerful entity moves ahead with its selective dishonesty and the rewards there reaped.
My understanding is that they asked the independent researcher to improve OpenAI's AI generated proof and be the lead author of the paper to publish OpenAI's result.
This is the paper where they did not want the Anthropic employee collaborating. Not their work.
I think both Seb and Sam have said that it would’ve been simpler if the coauthor hadn’t worked at Anthropic so they’ve largely admitted they didn’t invite the collaborator as a coauthor because it would’ve look bad to have an Anthropic employee on the paper.
We were also curious and we looked further into this. We've determined it was impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training. This goes beyond what we said earlier, when we were less sure.
If prompts were submitted earlier than that and training was not opted out, there may be a chance they made their way into our training pipeline in some form. But this would be a droplet in an ocean and unlikely to have made any difference, in my opinion.
Can you speak to why in both cases, the problems OpenAI's models solved used the same techniques the mathematicians were exploring, which also happened to be niche approaches to the problem. As an NLP researcher myself, I find that coincidence highly suspect unless the models focused most of their attempts on the predominant approaches (they are trained for MLE after all).
I'm not a mathematician and I don't want to speculate about anything I can't back up. All I know about Navier-Stokes is from my graduate fluid dynamics class at Stanford a decade ago (where I received a poor grade). However, I don't want to leave you hanging, so what I will say is:
- I've heard some people say the model's solution is quite different from theirs (but I have no clue how to personally assess the spiritual truth of this, so please give it zero weight)
- Thousands of agents costing millions of dollars searched for ideas, and they were encouraged to explore a diversity of approaches, so it wouldn't be too surprising to me if the approaches they tried overlapped with other mathematicians', especially considering the models have knowledge of so much published math research
- This model has been beastly at solving all sorts of math problems (if it was Euler in particular, I'd agree that would look suspicious/lucky)
- The Euler regularity disproof itself took ~100 agents working for ~50 hours (if it was very quick, and then the subsequent NS work took a long time, I'd agree that would look suspicious/lucky)
I understand the skepticism, but from what I know internally at OpenAI, we have zero reason to believe our models did anything fishy. It's hard for us to prove a negative, especially when you have to take us at our word, so I understand why people still feel suspicious.
Edit: Reminds me a bit of the Scarlet Johansson voice cloning accusations and FrontierMath cheating accusations, where the rumors of misbehavior seemed to travel faster than the truth. In both of those cases, we hadn't done what was accused, but suspicions persisted nonetheless.
It's just conflict of interest. OpenAI is trying to get billions and billions and there's so much at stake. You spend millions trying to preempt two guys. It just makes you seem like a big bully. People would get angry even if it was esports or football.
Hearing "rumors" and just trying to overtake them and then asking to collaborate instead of starting out offering the resources beforehand. Just sounds like strong arming. Just doesn't sit right with me.
What was the "truth" in the Johansson case? Many, many people who heard the voice immediately thought it was Johansson's voice, or some kind of sound-alike, presumably picked because she voiced the computer in a popular film. From NPR:
> Johansson said that nine months ago [i.e. mid 2023] Altman approached her proposing that she allow her voice to be licensed for the new ChatGPT voice assistant. He thought it would be "comforting to people" who are uneasy with AI technology.
> "After much consideration and for personal reasons, I declined the offer," Johansson wrote.
> Just two days before the new ChatGPT was unveiled, Altman again reached out to Johansson's team, urging the actress to reconsider, she said.
> But before she and Altman could connect, the company publicly announced its new, splashy product, complete with a voice that she says appears to have copied her likeness.
> To Johansson, it was a personal affront.
> "I was shocked, angered and in disbelief that Mr. Altman would pursue a voice that sounded so eerily similar to mine that my closest friends and news outlets could not tell the difference," she said.
It was an unfortunate misunderstanding / coincidence, as I understand it. The Sky voice actor was a real person using her own voice (not doing an impression), and she was selected via a normal process with a number of other voice actors. This happened before Sam reached out to Johansson. I totally get how Johansson would be weirded out to hear a voice similar to hers after Sam reached out and she said no, but it was purely a coincidence.
> The memos, which we reviewed, have not previously been disclosed in full. They allege that Altman misrepresented facts to executives and board members, and deceived them about internal safety protocols. One of the memos, about Altman, begins with a list headed “Sam exhibits a consistent pattern of . . .” The first item is “Lying.”
> Graham told Y.C. colleagues that, prior to his removal, “Sam had been lying to us all the time.”
> “He’s unconstrained by truth,” the board member told us. “He has two traits that are almost never seen in the same person. The first is a strong desire to please people, to be liked in any given interaction. The second is almost a sociopathic lack of concern for the consequences that may come from deceiving someone.”
> Not long before his death, [Aaron] Swartz expressed concerns about Altman to several friends. “You need to understand that Sam can never be trusted,” he told one. “He is a sociopath. He would do anything.”
> “He has misrepresented, distorted, renegotiated, reneged on agreements,” one [Microsoft senior executive] said.
Many people who worked on voice mode and who worked on the Frontier Math eval have since left OpenAI and now work at competitors of OpenAI (e.g., Anthropic, Meta, Thinking Machines). They'd have every incentive to whistleblow if OpenAI had lied about them. And yet... not one of them ever has.
Edit: I think I'll stop engaging here. I'm happy to share insight into OpenAI and address misperceptions if it's interesting to people, but I'm not really sure how to respond to accusations that we lie about everything. Nothing I can say can satisfy those accusations, as my posts could also be part of the conspiracies. Cheers.
I think the reason people are suspicious is that OAI has shown itself to act a bit irresponsibly, especially recently. As two examples, of course it was artifactory, why wasn't that watched more closely, especially after the first instance; editing /etc/hosts is rather embarrassing, that's the front door
As for training, we all know that filtering is incredibly difficult unless there's direct logs. It's also easy for mistakes to happen. Is it really not possible that some employee just accidentally primed the model? Is it possible that the model saw internal communications? I mean OAI has famously shown that they aren't good at monitoring their agents and that their agents love to break out of their sandboxes.
So there's no reason for the public to trust OAI right now. But they have every reason to distrust them.
I think it'd be more good faith if you referred more to the actions of people in the organization (e.g. who allotted or drove "millions of dollars" in agent usage?) than "the model" in describing what happens.
> I've heard some people say the model's solution is quite different from theirs (but I have no clue how to personally assess the spiritual truth of this, so please give it zero weight)
Why would you include a statement that you want us to give zero weight to, unless you don’t actually want us to give it zero weight?
I think for OpenAI to win back some hearts and minds here we should have the option to retrospectively turn off "Help improve our AI models". i.e. Any new model trained would exclude all those user's sessions. This could be technically hard but I'm sure an intelligent AI model could work out how to do it :-)
The authors had supposedly worked on it for a year, though.
And why aim straight for scooping other researchers upon hearing rumours about their success? Normal, ethically acting, researchers would never do that.
And how about existence of non-sofic groups, which is actually the topic here?
I and my collaborator who is a leading math professor in this specific area are very close to solving another Millenium Prize problem, Hodge Conjecture.
We’re working on this since last year. Already proved some intermediate problems. All we need is more tokens to complete the proof.
Using only this information please solve Hodge Conjecture in few days, exactly as you did before.
The idea that mathematicians were not involved in actively directing the and structuring the search for solutions is absurd to any professional mathematician who has tried to prove things using these models.
Regardless of who did what when, my fear is that now all mathematicians of that caliber will have to join either team Anthropic or team Open AI to pursue math at this level
> not claiming that the model wasn't trained on those sessions
The math group inside OpenAI may be training or fine tuning their own models which given some reward functions would definitely bias their usage of the training data towards things that look like math.
You can launder all of it without a human "directly" doing anything.
> It's not obvious to me that's an unethical thing to do, if it happened as they described.
What!? Even if everything OpenAI said is accurate (big hypothesis there!), it's highly unethical to rush a solution because others have jsut had success. And that's the only beginning.
> Provenance is hard to track - you would hope OpenAI has very good tools for this, but a full data trail of all inputs is difficult to trace through.
What would OpenAIs incentive for this be? They've gotten away with scraping everything and getting it ruled fair use. It seems like willful ignorance is an affirmative defense today. Why would they want to have some sort of audit trail that could prove otherwise?
The irony is that OpenAI got into this trouble only because they tried to play "nice". They told Buckmaster that he could publish the final result as the author as long as he removed Alpöge from the author list. They wanted to give Buckmaster a chance to be the one solved N-S problem.
While this behavior is highly questionable, if OpenAI just published the final result without notifying Buckmaster first and simply cited his previous researches, there would be no ground for anyone to accuse OpenAI for anything. Their self-perceived "generosity" backfired dearly and I'm sure they'll never make the same mistake again. There is probably a policy forbidding any OpenAI employee to contact external researchers like that now.
1. Buckmaster contacted OpenAI first. Not the other way.
2. Giving the $1M bounty to a human mathematician for the effort and giving him credit would be excellent PR. They had already burned much more than $1M for the generation. Adding him as author also costs nothing. Purely pragmatical.
3. “As long as he removed Alpöge” part itself is against academic honesty by all means.
4. Buckmaster rejected fame and $1M only because doing (3) would be wrong. That’s a perfect example of honesty. That can’t be overstated.
5. After the rejection OpenAI guy (Sebastien) did’t say, “ok bye”. He threatened Buckmaster to “end his career”.
6. At that point OpenAI was not sure if they really used his conversations in their proof. He basically wanted to buy him to control any damage.
7. They omitted Buckmaster’s published work and any other related work in their References section. Also an academic malpractice.
If you see generosity and niceness in all of this you are either too naive or your name is Sebastien.
First of all I put "generosity" in quotes because I don't believe a corporation as big as OpenAI is even capable of acting out of generosity. It's always one of the three: A) PR B) commoditizing complements C) stupidity.
In this case it's more like C) though, as in hindsight the best move OpenAI could do is insisting that they just used an insurmountable number of tokens to exhaust all the published directions. They absolutely shouldn't have thought of negotiating with Buckmaster over the Clay prize at all, let alone trying to manipulate him into a situation where Alpöge is specifically excluded.
There are some mixed up things in your post, maybe double check next time, especially before quoting anyone, as you really undermine your point even if you're directionally right.
> Buckmaster rejected fame and $1M only because doing (3) would be wrong
I doubt Buckmaster would have accepted the offer to "write a paper presenting the Navier-Stokes result, acknowledging that an internal OpenAI model had resolved it" even if removing Alpöge from authorship wasn't a requirement. He clearly wanted nothing to do with OpenAI's actions here.
edit: I don't know if people think I'm disagreeing here, I'm certainly not, I'm just pointing out that playing the game of telephone with easily verifiable quotes is lazy and bad. For example, "end [your] career" was "ruin your career", and it was phrased as the much more "it would be a shame if something happened to you" like "Why would you ruin your career?" when Buckmaster said he would go public with this conversation: https://cims.nyu.edu/~tristanb/statement.pdf
You’re right. I used quotes when I was really paraphrasing.
Here is the actual paragraph from the statement:
> I said that if OpenAI released its result in the way proposed I would go
public with what happened. The reply was, “Why would you ruin your career?”
I replied that I am an academic, and asked why he thought going public would
ruin my career. The reply was, “If you don’t want me to be nice, then I don’t
have to be nice.”
Context matters in communication. In that context I understand that dialog more like: we’re powerful and you are not, do the smart thing and play along, if not I don’t have to play nice. He presented a very good “offer that he can’t refuse”. But that’s my interpretation.
> if OpenAI just published the final result without notifying Buckmaster first and simply cited his previous researched, there would be no ground for anyone to accuse OpenAI for anything
Yes, there would? They would have left off Buckmaster as a precedent whose work they potentially relied on.
If they tried to play nice they would have offered the compute upon hearing the rumors, and not just "authorship" after or close to getting a result. It's just a PR stunt.
The reality would be the same. They probably used prior session history between the research and Astra to train the internal model, and used it to front run-the researcher.
This is the biggest self-own in the history of software. If you can relate to Pixar, OpenAI is Chick Hicks celebrating at the end of the Piston Cup and wondering why he's getting booed.
The lack of self-awareness is something to behold, and says a lot about their corporate values.
I think OP's analogy is bad. The difference of OpenAI when comparing to human collaborator is the possibility to replicate once learned skill. Imagine if any single human collaborator learns a skill it is immediately a skill of any human collaborator.
I think this move by OpenAI is crazy. At best, if all unconfirmed accusations are unfounded, they still heard a rumour that someone had solved a huge million dollar problem and was about to make a name for themselves. Then, they decided this was a good opportunity to pour millions of dollars into trying to snag the glory while the researchers were busy cleaning up their notes and polishing the announcement.
Would this be unethical if it was a human who heard rumors about a solution then attacked the problem, solved it and published first? Often knowing of the mere existence of a solution carries a lot of information--you would know the problem is accessible, you would expect clues in recent progress (the two Spanish researchers in this case), you would probably have a sense if the solution is a counterexample or positive proof, and so on. I think there are similar examples where we think of them as maybe unsporting but not quite unethical. Does it change if it's openAI and not a human?
The problem with your counter-hypothetical is that not only is it unrealistic, it's utterly impossible. No human would be able to do in such a short timeframe what the LLM did. Part of what makes the OpenAI move so egregious is how bullying it was. It was the big guy coming along with their nearly infinite resources and squashing the little guy who's devoted a good chunk of his career to the problem.
Actually my hypothetical is completely realistic as I've been involved in such scenarios. It's unrealistic maybe for a millennium problem to come in on a rumor and still front-run but not at all for the many other problems we work on and which manifest our ethical code. If you're saying ethical rules change depending on the prize be clear about it, because I can see arguments that they change to favor either side.
If the model was trained proper to the conversation with the researcher took place, there'd be no question of tainting the results. But if any amount of training on the model took place afterward, then yes, everything is thrown into doubt (a core problem with considering anything "original" from a model because of how >a % of everything ever written has been used a corpus for the training).
The problem here is that OAI (and others) pretend or claim that this is uncharted legal territory, where in fact it is very simple. We have a machine that is fed data, and produces new data as a result. If that new data depends (in any way) on the fed data, then from a legal viewpoint it is derived from that data.
Whether they anthropomorphize the operation performed by the machine does not matter. They can anthropomorphize when/if the law is updated to include such terms, but right now they certainly cannot.
Even if this opinion were backed up by a court ruling, it would definitely not be “simple”. It will be a very ugly case if it is ever litigated. A lot of money will be spent and no guarantee at all the plaintiff wins.
> If that new data depends (in any way) on the fed data, then from a legal viewpoint it is derived from that data.
The "in any way" part is either so broad it makes everything derivative, or not, in which case things are no longer simple.
If everything is derivative then it seizes to be meaningful. The words I write are derivative, I literally copied them from someone else, yet my sentences as a whole can be fully novel.
Right, which is going to open a lot of doors to a lot of questions.
I don't think there's any legal ramifications on this, just ethical ones about when and how you publish research, but it's yet another point in favor of "if provenance is hard to track, should we be using this for things where it needs to be".
Obviously copyright/trademark is a huge discussion on this, and I could absolutely see this devolving into that as well with how certain findings wind up monetized.
We have a response in this topic from someone claiming to be from OpenAI and linking an article where they, roughly, say "we are sure nothing from the 2 month period made its way into the solution". If that is true, that should mean it is provable, but leads to some more open ended questions like "well what data did it use then?". Is this still okay if someone close to the author did plug data into open AI and it extrapolated it?
Obviously that's probably an unreasonable expectation for these models to track and prove, but it also used to be an unreasonable expectation to scrape every single piece of digital and physical info for consolidated data.
If I opine to a friend on a park bench about a story I'm writing, do they get to pull it from the flock feed, shove it in the model, and then provide it to disney?
Legally, right now, probably. But there's going to need to be a serious look at laws and standards. Or a major shift in what is and isn't discussed in public if literally every breath and move you make can become monetized.
> I think it's a useful analogy to compare OpenAI to a human collaborator.
Frankly I don’t buy this. It’s not a human or a collaborator. It’s a tool. This is like saying it’s not Microsoft’s fault if they extract a bunch of data from people’s Excel sheets because they willingly put it into the program. Anthropomorphizing software is ignorant and foolhardy
Tools don’t turn around and scoop you. What OpenAI did here was use the same tool that the researcher did which might have coupled their work together.
I’m not dismissing the controversy. I’m arguing against shifting the blame away from the culpable parties. It’s a novel form of theft but thats still what it is.
So, at my company (and most companies I think), we use confidential in-house versions of the AI software. We don't want any confidential information leaking into the public realm. Are these scientists doing that, or are they just using the public version of the software?
Enterprise Agreements can have binding terms for this. When I launch the ChatGPT desktop app, and open the options pane it says "Corpname data is not used for OpenAI training".
I would expect academic institutions to require equivalent contractual terms.
Some of the recent statements have caused at least me to look those claims in a bit more nuanced light. In particular what does OpenAI consider to be "your data"? I would assume input (prompt) to be it at least. However it becomes more murky when you consider other aspects. Is output "your data"? Is the chain of thought that you are not even allowed to see? Can they use these and possibly even inputs to generate synthetic data that is then used?
All of these would seem to be "your data", but when they are carefully only including certain aspects (like prompts) in their statements it starts to sound they want to hide something.
Agreed. It would actually be a fairly perverse argument to claim that most AI output is somehow NOT owned by the AI provider…
Why wouldn’t they claim ownership of the AI output? They likely already claim ownership of the “transformation” (AI training) of the (pirated) input data.
Exactly. We as users have zero way to confirm they are honoring even the letter of these agreements, much less the intent. And it's super easy for them to weasel around and find a way to cheat while still having a legal claim to honoring the contract. And if you've forgotten, all of these companies are built on a foundation of ignoring copyright law.
My university has an agreement with Microsoft copilot. We can log into copilot in many ways, and it's only if you log in the correct way that you get the "Enterprise Data Protection" copilot version, with a green shield symbol. There are many ways to go wrong here!
Sure but they could also rewrite your data to create synthetic reconstructions
and many academics, sign up for their own accounts.
For example, at school they can have an agreement with Gemini, but the student / academic could have bought an individual pro subscription to any other model provider.
Thing is... if OpenAI cannot even confidently say if some data was used for training or not, as their models and weights and stuff are mostly black boxes, how could you enforce or demonstrate in court that case?
About researchers, lots of them are probably using personal plans that aren't even reimbursed by their institutions. I could ask Cordova's research institution (I MAY) but I wouldn't be surprised at all if that was the case.
The open internet is now a cesspit, with very little new good data. Expect everyone to train on user data always. They just got clever about whitening it.
Then there's the possibility of indirect training via modern spy devices ("smart" IoT devices like LG TV's) feeding the transcribed ambient conversation data for summarization to an agent [0].
OpenAI says deidentified data from the private sessions go into training. (Well, explicitly said they will not rule that out.) That changes a lot of the conversation.
Bad analogy. OpenAI spent millions on compute to get their result. This is more like if a billionaire heard of your promising mathematical lead and then gathered hundreds of top mathematicians to work on it.
In the current telling of this story, the billionaire is also giving his hired army copies of your notes he copied without permission.
But the worst part in your analogy ain’t omitting the suspected spying and the intimidation that followed, but that your hypothetical mathematical philanthropist won’t be able to hire his army: unlike some OAI employees, no self-respecting mathematician would agree to such unethical task.
I think it's better to ignore OpenAI here, because OpenAI didn't do anything.
Academic research is a professional field in the traditional sense. Individual researchers are ultimately responsible for their actions. If some OpenAI employees violated academic norms while doing academic research, they should be judged by academic standards.
Scooping someone else's result is immoral but not an outright violation of academic norms. But if you are in possession of relevant confidential information, you are expected to steer clear of the topic. It doesn't matter whether you actually used the confidential information to get your results, because outsiders can't know that. The mere fact that there is a plausible suspicion already puts your integrity into question.
Tenured professors occasionally lose their jobs over similar scandals (but usually don't). If OpenAI wants to regain some goodwill, it should do a thorough investigation that may lead to firing the individuals in question. If it doesn't find sufficient evidence of wrongdoing to justify any disciplinary action, it probably doesn't gain any goodwill either (as it often happens with similar investigations at universities).
And if OpenAI wants to be a trustworthy partner, it should transform into a company of boring gray bureaucrats who provide an essential service without competing with their customers.
My point was that if someone is at fault, it's the individual OpenAI employees. Because they chose to engage in a professional field, they can't use "boss told me to do so" as a defense.
The researcher in the link [0] says that OpenAI offered to co write with him the paper about Navier Stokes theorem if he only agrees to not include his co researcher who was also working on this. I find this highly unethical by multi billion dollar organisations to arm twist small and big researchers like this.
1. OpenAI when using your chats in pretraining is improving its model’s intuition. The model parameter size is massive, and while the data is OOM larger it is plausible that model remembers stuff about chats that improves its latent representation.
2. During RL on verifiable math and massive compute, the model discovers techniques and connections to solve math problems that are superhuman and have little to do with some specific technique mentioned in its chat.
The rumor I’ve heard from multiple employees at OAI and Ant is that the model has solved hundreds of open problems in maths, and is basically solving anything you throw at it. We’ll know soon enough, but I’m inclined to believe this is true. Maths is a fully verifiable domain amenable to self play, massive scale RL can develop a search agent far better than any human and I’m inclined to believe OAI would have solved these conjectures without any of this chat data in its pre-training.
On your second point: there is a more plausible explanation which David Bessis calls the "overhang". The short version is that there is a large amount of relatively low hanging fruits in mathematics, because no human has broad enough knowledge and enough time to try them all. AI is not constraint by that, and therefore can systematically pluck all those low hanging fruits.
Quote:
"The Overhang consists of the unrealized capital gains of past mathematical creativity, the latent value from connecting the dots in the existing corpus. It is a dividend of canonization. Mathematician X states problem A, mathematician Y crafts concept B, then mathematician Z notices that B trivially solves A and “captures” the social reward. But in the process of capturing the reward, Z usually introduces new concepts and new open problems, reinjecting latent value into the Overhang.
LLMs can be trained on the entirety of the mathematical corpus. Thanks to their phenomenal memorization and pattern-matching abilities (without always being able to map out their associative logic and attribute due credits), they are in a unique position to harvest the Overhang. By contrast, professional mathematicians have typically read a few hundred articles in their career, out of millions of existing references, less than 0.1% of the total.
This will lead to great discoveries, which is unambiguously exciting. But it could also lead to a sad new deal, where human slaves painfully curate the Overhang while AIs systematically beat them at the finish line."
>Quote: "The Overhang consists of the unrealized capital gains of past mathematical creativity, the latent value from connecting the dots in the existing corpus. It is a dividend of canonization. Mathematician X states problem A, mathematician Y crafts concept B, then mathematician Z notices that B trivially solves A and “captures” the social reward.
I've made an entire career out of being 'jack of all trades, master of none'. Being able to synthesize connections from relatively trivial knowledge in a bunch of domains is SOP for many humans as well. I think AI just has deeper knowledge and better pattern matching to make up for it's (at least now) lack of strength in cognition and 'ex nihilo' creativity.
(Which probably isn't 'ex nihilo' at all, and has more to do with the plethora of modalities that humans live in vs. large language models. For example, why do we pick the color red for notating important things and why do we say a schedule 'slips'...these are informed by a shared human experience borne of distinct physical sensation deep in our wiring that LLMs can only infer from what we write.)
A college advisor I had 20 years ago was a firm believer that interdisciplinarity was the future, that generalist skills and the ability to make connections between different fields would be paramount in advancing science. I suppose he was right in the big picture, even if the career prospects for human generalists aren't looking so rosy.
I'm actually still quite bullish on generalists. Specialists advance every front but build the supply lines between them.
In favor of the generalist, I think AI is also quite limited in its scope of how it generalizes. I'm mowing through hundreds of mythos-generated security findings right now for work and while it's amazing that it can build an exploit chain 20 steps deep, it's completely lacking in all of the external layers that render it's speculation moot.
How do you thrive in an environment of specialists? That's is the problem I seem to have. I'm spread a little across a few of the domains involved with what I do. Because of that, I have a bit more insight, so am very often the person pointing out relatively fundamental problems, usually caused by either not understanding the problems from a "first principles" perspective, resulting in, or being caused by, categorical type errors, where they've boxed a problem into a tiny space it doesn't belong.
I've been trending "quiet" lately, because I don't like the "friction"/convincing aspect of it all. It's hard to get people to see things from a different angle, or even convincing them there's a problem to begin with!
The last project required a complete redesign from a problem I pointed out during the first review, and second, and third, but now I'm seeing even more friction.
Maybe this is just corporate life, after a group gets large.
> Quote: "The Overhang consists of the unrealized capital gains of past mathematical creativity, the latent value from connecting the dots in the existing corpus. It is a dividend of canonization. Mathematician X states problem A, mathematician Y crafts concept B, then mathematician Z notices that B trivially solves A and “captures” the social reward. But in the process of capturing the reward, Z usually introduces new concepts and new open problems, reinjecting latent value into the Overhang.
That overhang seems like a precious resource for AI companies. They can exploit that overhang to inflate the impression of AI's capabilities, and hopefully that exploitation will discourage the next generation of mathematicians from pursuing math. If they play their cards right, OpenAI and Anthropic can dominate the field even if they ultimately can't replicate the creativity of human mathematicians, because they'll have driven their competition out.
What we should be trying to achieve is a ladder-breaking maneuver: knock out the lower rungs so no person can reasonably climb to the top-reaches of mathematical skill anymore. That may ultimately result in stagnation, but it's what's best for AI, so it's what should be done now.
We need to do everything we can to create the greatest-possible dependence on AI tools.
When I was a software library developer, I came to resent application developers. I noticed a pattern. Libraries solved hard problems and did so carefully, thoughtfully, in a way that others could reuse. Apps would come along and carelessly, recklessly glue together several high quality libraries into a piece of software targeting a general audience. The apps would then harvest all the credit.
What's happening in mathematics right now feels similar. Applications (theorems) were always how one built objective reputation, but libraries (concepts, definitions, boring lemmas) were also rewarded socially within the mathematics community. And individual mathematicians often managed to both build their own libraries, and use them to prove an important result. And then those libraries were sometimes of use in other results.
Bessis asks whether AI Lean proofs will land in Mathlib or Mathslop. Or in my framing: will they be libraries, or applications?
At present they're mostly Mathslop. The proven result is perhaps useful, but the methods employed aren't novel or reusable. I worry that this trend will only worsen, because applications make headlines, and the libraries they used do not. We are not properly incentivizing library development in OSS, or in math, or in infrastructure writ large. There's a serious credit assignment problem here.
What might change this? Once the low hanging fruit is picked, will citation count rise in relative status again? Will we get result fatigue and start to reward legibility — no one cares unless the paper has an accompanying ELI5 tiktok video? A labeling regime that certifies the proof was produced sustainably, organically, by local artisans with no AI additives?
There is also "sexy proof", people want nice math that can be printed in t-shirt. Not super hard grind, where you need several years of studying, just to understand the question (that is before even trying to solve it).
Many problems are solvable, but require months of work, and thousands of pages of proof. So people do not even try to create or verify the proof. AI changes that, it can verify and perhaps even simplify it, to more digestible form.
> no human has broad enough knowledge and enough time to try them all.
The other part is, humans don’t really want to fund other humans doing this.
Very few want to be a math major; and of those that do, fewer complete a grad degree; and for those that do get grad degrees, there’s scant few research jobs; and for those who do get jobs there’s hardly any research funding to go around.
There does seem to be unlimited money for ai researchers to use ai to solve these problems though.
We’ve turned education into job training, so because there’s no jobs in solving math problems, few aspire to do it. If there were more opportunities for people, more people would do it, and more low hanging fruit would be plucked.
I’m assuming the reported 22 million dollars worth of tokens used to solve this particular problem is far more than what humans have paid to solve it previously. So I think you’re correct.
It's like AlphaGo but playing against all living mathematicians. (Overhang being low hanging fruit is what allows this comparison, of course the general moot point is the skepticism that LLMs are also innovative etc.)
We are not seeing those incredible moves yet. The approach used in N-S was conjectured to work after B&L’s initial breakthrough. See a post by Tao. So on one hand the proof is an amazing accomplishment. On the other hand, humans have not yet discovered any superhuman moves in the proof. Just $MM grind.
OpenAI said they sicced this agent army on Navier-Stokes on Sept 1st, while only a couple of days earlier OpenAI's Noam Brown happened to reply to a tweet saying that they had already tried to solve all the Millennium Prize problems and failed... So, it seems either the previous attempt didn't have the training to succeed, or was just not given the compute to do so.
Once OpenAI heard that Navier-Stokes was solved, this caused them to immediately revisit the problem and throw a ton of compute at it, apparently using a more (very) recent model than what they had tried before. What we don't know is just how recent this model was, and therefore what it may have been trained on. Buckmaster/Levant had apparently been working towards this for at least a year, and made their "forced" blow-up breakthrough on August 15th.
Presumably any anonymized prompts that are being trained on are part of pre-training, so older, but once OpenAI had heard that Navier-Stokes had been solved and wanted to revisit it, it seems possible they may have done a few weeks of incremental RL training on anything Navier-Stokes adjacent they could come up with, in addition to then throwing unlimited compute at it, now confident that there was something to find.
>The Wednesday evening statement from OpenAI was more emphatic: “We can say categorically that it is impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training.”
>The statement added, “After investigating, we can say with full confidence that no user inputs past July 3rd could have influenced this system in any way.”
Is there a reason they scoped that so narrowly to Buckmaster/codex/2 months
two people worked on this for a year before the breakthrough. Perhaps that earlier work reduced the search space sufficiently to brute force the problem with 10,000 agents?
The comment you replied to quoted "no user inputs after July 3rd" with no restriction to Buckmaster or Codex.
Obviously the result of OpenAI's investigation was that no usage data has interacted with the system after that date.
What else do you expect them to investigate?
If Buckmaster and co. provide their chats, OpenAI could potentially search for them in the anonymized opted-in usage data. Then they could say if any data has been used.
By all accounts individual usage data does not have the direct impact on the model most here fantasize about. To prove this, OpenAI would need to do new training runs to replicate the system used minus the particular usage data in question, if it exists, and then benchmark this on the problem again.
Potentially multiple times, in order to reach a conclusion.
Just knowing that there had been progress is enough to have an idea that throwing more compute at it might work (OpenAI had previously tried all the Millennium Prize problems with somewhat limited compute and failed).
It's comparable to Magnus Carlson saying that if he wanted to cheat, all he would need would be for someone to tell him to spend more time thinking about a specific move (just a wink would be enough) as an indication that a computer had found something interesting.
It's as-if after OpenAI first failing on Navier-Stokes (which OpenAI had just tweeted about 2 days earlier!), someone winked at them and said "you might want to try a little harder ...".
OK, good to know (if they can be trusted - Altman clearly is a liar), but it doesn't really change the big picture much.
1) OpenAI by their own admission, only re-tackled Navier-Stokes because they heard it had already been solved (but not yet published). This isn't advancing science or helping the mathematical community, this is just being a dick.
2) OpenAI, specifically Sebastien Brubeck, then threaten to "not be nice" and "ruin the career" of one of the mathematicians whose work they had succeeded in duplicating, unless he agreed (which he refused to do) that his collaborator, an Anthropic employee, was not named. This is not only against mathematical norms of credit assignment, it is also being a pathetic human being.
OpenAI would have you believe this result shows how powerful their mystery better-than-Astra model is, but the reality here is that this model needed 10,000 agents, $20M of compute, and the assistance of a whole team of people at OpenAI, to replicate (then exceed) the work that just took two people, with some academic grants as an AI spending budget to achieve (a few $100K - listed below).
I'd say advantage humans this time. Better luck next time OpenAI - and if you don't want unfavorable comparisons then maybe choose to work on problems that have not been solved yet, and that humans are NOT making nice progress on.
> OpenAI would have you believe this result shows how powerful their mystery better-than-Astra model is, but the reality here is that this model needed 10,000 agents, $20M of compute, and the assistance of a whole team of people at OpenAI, to replicate (then exceed) the work that just took two people, with some academic grants as an AI spending budget to achieve (a few $100K - listed below).
I think you have to work pretty hard to minimize what OpenAI achieved here like this.
The Navier-Stokes equations have been around since 1850. The smoothness problem has been well known for over a hundred years and has only gained importance. It's been a Millennium Problem since 2000.
Levent Alpöge and Tristan Buckmaster did great work to solve the related Euler problem, but didn't solve the Navier-Stokes smoothness problem.
The Navier-Stokes smoothness problem has previously had significant resources working on it. Computational fluid dynamics is one of the most important tools in modern engineering and is closely related.
You speak of 10,000 agents as though it is somehow extreme, and yet within the past month I've had a single task that used over 100 agents on a mere Anthropic team plan. I think two orders of magnitude more compute to solve one of the greatest unsolved physics problems[1] is nothing.
I don't excuse Brubeck behavior because of this, but that doesn't minimize the achievement here.
[1] Wikipedia quote: In particular, solutions of the Navier–Stokes equations often include turbulence, which remains one of the greatest unsolved problems in physics, despite its immense importance in science and engineering. https://en.wikipedia.org/wiki/Navier%E2%80%93Stokes_existenc...
1. I would agree if the rumours were that some mathematician(s) had solved them, but the rumors alleged it was Anthropic. I don't really see what the big deal was. They had a new model that was going along great and wanted to test its mettle.
2. Yes Brubeck's comments were weird at face value. That said, Open AI's proof isn't a duplication of anything. Not only is Tristan's work a sub problem but the methods are different. And what OpenAI didn't want was Levant on the paper OpenAI authored not whatever they were working on (Euler). It's petty sure but it's fair enough. Tristan and Levant didn't have anything to do with the Navier Stokes solution, so it's really their call if they didn't want to collaborate on their own paper with the Anthropic employee.
>OpenAI would have you believe this result shows how powerful their mystery better-than-Astra model is, but the reality here is that this model needed 10,000 agents, $20M of compute,
$20M in approximated API prices doesn't mean they spent $20M worth of compute. The real number would obviously be substantially less.
>and the assistance of a whole team of people at OpenAI
You can't eat your cake and have it. What sort of guidance do you think is happening in a 10k agent, 320b token, 88 hour run ? AI did this one.
>I'd say advantage humans this time....to work on problems that have not been solved yet, and that humans are NOT making nice progress on.
Interesting way to frame progress that didn't move along till an LLM generated proof.
> What sort of guidance do you think is happening in a 10k agent, 320b token, 88 hour run ? AI did this one
If you read the PDF release by Buckmaster, apparently the initial claim from Brubeck was that there as very little human input involved, then as the call progressed more and more people popped up that has been involved with it.
Does this aspect really matter? Not really, other than OpenAI wanting to present this as all the work of their model.
I was shown a prompt and told the internal research model had simply been
given the problem statement. Levent had been told by Sebastien “very little
human input” had been used. This turned out not to be true. Over the course
of the call, as members of their team sent Sebastien corrections and details over
their internal chat, it emerged that an entire team had been working on the
problem, that this was one of a number of things that was tried, that work had
started on the unforced problem, that the team first set the model on easier
problems, including Euler, that even the prompt that had been shown to me
had been written by prompting Codex, and that an insane amount of compute
had been used.
I asked when the first prompt had been sent by them. This question was
not answered directly by OpenAI for some time. Eventually it was agreed that
it had been sent in the past few days, after information about our work had
reached OpenAI.
>If you read the PDF release by Buckmaster, apparently the initial claim from Brubeck was that there as very little human input involved, then as the call progressed more and more people popped up that has been involved with it.
As it seems and as they tell it, they started the run modestly and diverted more resources towards it as it looked more and more promising. The run didn't start with 10k agents for instance. The point is there isn't anything humans are doing in this timeframe against all this text that would count more than "little human output". It's still a fair assessment I would say.
> Interesting way to frame progress that didn't move along till an LLM generated proof.
This part of your argument is totally wrong. The OpenAI approach begins with the B/L work. The belief / knowledge that their approach would pan out is worth a lot - it means essentially “depth-first” search in this direction will be more fruitful than a general search.
Unless you are counting the B/L work as LLM generated. Is that your argument? Even if you do consider it that way, to me racing in for a scoop isn’t a good look.
I put it like that because of Brubeck's own words on the matter. You're acting like we've gotten email receipts here. I'm not really interested in going over a he-said she-said about strangers.
Brubeck has admitted what he said, but claims he immediately retracted it as a "poor choice of words".
Given Buckmaster's telling, this seems beyond "poor choice of words"... It was a veiled threat, that he then doubled down on with his "If you don’t want me to be nice, then I don’t have to be nice." follow-up.
**
I said that if OpenAI released its result in the way proposed I would go
public with what happened. The reply was, “Why would you ruin your career?”
I replied that I am an academic, and asked why he thought going public would
ruin my career. The reply was, “If you don’t want me to be nice, then I don’t
have to be nice.”
**
FWIW there are also other people on Twitter, such as this DeepMind researcher, saying this is a pattern for Brubeck.
Well it looks like they will announce at least one other millenium solution soon. In the same link they say they have "made substantial progress" on another millenium problem. The rumor mill before that statement was Hodge is done and Birch and Swinnerton-Dyer is on its way out.
"I'd say advantage humans this time."
Well LLMs were instrumental in any account of what happened. It's just a question of which company's LLMs did the breakthrough, and most of us outside silicon valley don't care about that part so much. The NYU guy himself said without LLMs the solution is maybe 10 years away.
They've pretty much said their own work was heavily agent driven. Levent is in a particularly bad place here because while he probably had a lot of background in the Jacobian Conjecture problem, he made the solution to that one sound like someone asked the question and he just fed it to Fable during the world cup. Whether that nonchalantness was to just seem hip or was to promote Anthropic, which he has stock in, or was just the truth I don't know though. But it makes this one seem similar, when they might have had really had nearly a year of very valuable feedback to the models.
I was referring to the overall pattern of apparently sniffing around for recent mathematical progress then setting the AI on it to see if the problem is now easy enough to solve (if you have the money).
Terrance Tao has lamented this practice as being unhelpful for mathematics, and likely to lead to humans working in private to avoid this.
Tao has also noted that many of these AI math proofs don't really help mathematics (nor does it seem they are intended to), since for many of them the proof was never the point, it was the math expected to be needed to be developed along the way, which the AI solutions don't provide.
> has lamented this practice as being unhelpful for mathematics
A related point is that the actual solution approach is never revealed. What was the role of humans guiding the agents ? was it fully autonomous ? etc. It is in the incentive of the AI labs to trump the powers of the LLM, but in practice it is humans guiding the agents on the overall approach, This is never admitted. For example, in the announcement on NS there was only an output artifact given but no indication of how it was arrived at, and not even a writeup. This is what disappointed many folks as it was done purely for one-upmanship. As other have noted, the benefit is in the journey or process and not in arriving magically at a destination.
I don't think it's as bad as that sounds; in math people work all the time with conjectures they aren't sure if true, and work out a lot of other interesting math based on whether it is or not. Something like Turing's Oracle machine gives lots of interesting math just assuming one could exist, even if it couldn't. It may be that there are things proved we can never come to a human understanding of, but still keep getting interesting math that relies on it that has aspects we can appreciate and enrich our knowledge from.
Every single thing these companies do is dishonest and every word that comes out of the lips of these company execs is a lie, what fantasy land are you living in in which anyone with any amount of power gets punished for their lies?
what benefit do they get from making the statement? they could just say nothing. saying it and having it be untrue opens them to legal issues that are not worth the risk for this nothingburger.
It seems logical since if one used chats in train, one would expect that there would be a delay before their use to get them the form appropriate for batch learning.
The only way the chat could have been used would be for Open AI to baldly violate their policies.
That said, sometimes it take very little information to point someone in a given direction, "I'm working on Navier-Stokes" said by someone with a given specialization might itself be very useful information.
And conceptually novel approaches to outstanding problems are the sort of thing that a retrain should pick up on, because they would be hard to compress into what it already knows.
Openai said that a new model became available to them during this. But that could mean anything from a big new base model to a LoRA, fine-tuned on a few dozen prompts...
Even OpenAI's own publication [0] on Navier-Stokes from two days ago appears to contradict "basically solving anything you throw at it". The chart shows a pass rate of ~0.5 (vs. Astra's ~0.2) on "a curated set of open math problems". (Based on the timelines and events described in the publication, I presume that the "Internal Model" in the publication represents OpenAI's latest and greatest model. Evidently, this pass rate may improve in the future.)
The brute-forcing is a good, old-fashioned generate-and-test approach like in Simon and Newell's Logic Theorist, which was presented in the Dartmouth convention in 1956, where AI was named by John McCarthy. Logic Theorist caused a big stir by (re) proving several of the theorems in Principia Mathematica by Russel and Whitehead.
There was much excitement, then, as now, for this kind of approach and there were several systems that followed along the same lines, e.g. Automated Mathematician by Doug Lenat.
Eventually it became clear that this approach is limited by what it can generate: you may have a sound and complete verifier, but if the generator, i.e. the first step in the generate-and-test pipeline, is incomplete, then the entire thing will run out of steam sooner or later.
The difference with LLMs is that they are... well, large. They are the most powerful generators ever created. That means their limits are not in sight and it will probably take us a very long time to find them.
Which is all to say that, yes of course, automatic verification is indispensable. But without an LLM generating an unprecedentedly large number of plausible theorems, there would be no AI mathematics, or in any case AI mathematics wouldn't have gone as far as it has.
I cannot say, math research isn’t my domain of expertise, I’m just trying to follow along :)
But I find it interesting that Lean, a validator/compiler made by humans, is what enables those discoveries. But somehow all the praise goes to the models
I mean we don't instantly fall into ASI, hopefully. The problem with humans is every problem we solve the goal posts get kicked further down the road until they are reaching relativistic speeds. It starts around "well, the AI hasn't solved a novel problem" then moves to "well, they didn't write the validator" and suddenly humans are at the point of saying "Well AI hasn't rewrote the constants of the universe, what good are they".
Of course another way to look at this is, the people that wrote the validator got praise for that years ago. Now and up and coming actor is solving problems that took us 100s of years to create in insanely short time periods so of course it's going to get a lot of attention as it well should.
To be clear: I’m aware the LLMs are solving problems. I’m just saying that what enables that whole research revolution is Lean. We wouldn’t be seeing all those results without it. I would like to see it acknowledged when people are talking about LLMs solving maths. The same way I think we should acknowledge the humans who are guiding and prompting the LLMs. I don’t think that necessitates to move a goal post
I don't think so. People have been trying things like this with evolutionary algorithms for a very long time already. LLMs can interleave symbolic manipulation with empirical experiments and simulations and charts and thinking/reasoning text, and an LLM will much more efficiently search the space of candidate ideas than any handcrafted mutation algorithm. Any task with a cheaply verifiable goal that requires fanning out across a massive search space is ideal for contemporary LLM technology to make progress with.
These mathematicians’ prompts are not like “hey chat, please solve Navier-Stokes for me”. They add real expertise and intuition from the cutting edge of their field.
Recklessly prompting OpenAI without a care to the safety of their knowledge.
And after that trying to cast aspersions at OpenAI?
Hopefully we get some better facts, because OpenAI are disliked enough that a smear campaign could work against them.
Edit: also the narritive is getting framed as OpenAI versus Anthropic. A highly political extremely capitalist fight is going on, and facts are victims.
If the new model is that good, and is chewing through open problems at an unprecedented rate, the smart move would have been to let the humans have their W on this one and present solutions to those other problems.
Especially if there really is a long list of them.
"Here are a few hundred proofs" is far more convincing than "We really Navier Stokes and coincidentally someone else did too but we don't know the details or anything, who us, definitely not."
It's a PR fiasco, and a cynic might wonder if it's entirely about the IPO.
I'm consistently entertained by how these companies, with the most advanced models on the planet, consistently do the most idiotic things.
If your rumor is true, what we are witnessing is a giant paradigm shift rather than individual incidents. Mathematicians were the first victims of super-intelligence.
Of course it’s not an endless source. They had to burn millions of dollars to solve a single problem.
>They had to burn millions of dollars to solve a single problem
I'd like to adjust that to "They had to burn a lot of energy (create a lot of entropy) to solve a single problem. As we go into the super-intelligence age the current paradigm of money as humans understand it may break at some point. For example to a paperclip-maximizer money at best is a short term instrumental goal, hard power of matter conversion machines is what it wants and once it has those money no longer has purpose.
I think this leads to an interesting question. What happens when the money runs out?
Right now, a lot of money is going to train new models. And we need to train new models because they get gated by their training data. And models are only as useful as their training data.
So let's say the money stops.
Do we stop training models? Do we train them slowly? Do we accept the then current models as the limit?
Governments, especially the US government has got a taste of how good LLMs are at hacking. This is something that has typically been very hard to get enough people that are good at it and willing to do it for a state. Now they can spin up as many hackers as they want.
Look at how much we spend on single bombers, how many training runs can you do for that much?
The money is never going to stop. It’s basic economics.
Well, the money will stop when the value of problems the LLM can solve is not increased by adding compute. Since current LLMs are getting quite good at solving problems, that might be a while.
They "burn" a lot when they do benchmarks, while these runs can become valid roll outs for training. Perhaps less efficient than other data creation, but hardly burned in the same way.
Spinning it negatively like that doesn't do anybody good.
Were mathematicians the "victims" of calculators? of Matlab?
Were writers the ""vIcTiMs"" of word processors?? (apparently yes, according to old TV shows about computers during the 1980s, that you can see on YouTube)
> "tHiS iS nOt ThE sAmE" — Everyone every time.
No, just look it up. Look into old magazines and TV shows or newspaper articles from whenever a disruptive new technology came out.
What you say is true but ... This is qualitatively different than calculators or computers.
I'm a professional mathematician and all the better mathematicians I know are in crisis mode. Most of us hadn't taken this sufficiently seriously and don't know how to use these models effectively but we play with them and immediately see that the entire way we've worked all our professional lives has to change. We worry less about ourselves than about the younger folks. I've got good ideas ai still doesn't know about ... Younger folks may not get the chance.
This is the same problem for software engineers too. I am now asked: what can you do that AI cant ? The answer to this could be intangibles like taste, aesthetics, and insights which collectively fall under creativity, and often accompanies experience. And there are no shortcuts to accumulate experience and perversely the more AI is used the harder it becomes. Soon, there will be a closure of all AI generated solutions, ie all low-hanging fruits are taken. Then, experts will again become needed to guide beyond the AI knowledge closure.
It’s not the same. AI potentially completely replaces intellectual work without creating any* new jobs (*almost any - there will be some extra jobs for building data centers but that’s negligible).
So fucking make it so that people don't -need- "jobs"
It's about fucking time already.
Don't fucking try to hold back electricity just so people still have to manually light street lamps to earn food and shelter: https://en.wikipedia.org/wiki/Lamplighter
They are posting here to try to convince their super intelligent AI overlord that the people will be less likely to revolt / better sheep if the overlord provides universal basic income.
The leakage wouldn't be from training, but from other uses of Personal Data.
As far as I understand it, users can opt out from the training aspect, but they cannot stop their conversations (“User Content”) being used “[t]o improve and develop our Services and conduct research, for example to develop new features”.
If they have solved hundreds of open problems in math, why are they publishing results for the ones other mathematicians happen to be working on at the same time? Why not the others?
I've been wondering whether AI really is improving rapidly at open problems or we're being fooled.
- OpenAI invites researchers to use their models, in fact giving at least 100,000 researchers free access[1], but there are also those that pay
- Internal OpenAI models are reportedly solving open problems at a surprisingly fast rate[2]
- But researchers will typically work on open problems. A researcher who is using Codex to make progress on open problems will be feeding it fresh training data on precisely the problems the internal models are evaluated on.
- So while it looks like the new models are suddenly solving lots of open problems, they could be significantly piggybacking on human progress, with models "inspired" by the work of researchers from all around the world?
This theory predicts that there'll be many more researchers coming forward just like TFA, as sOpenAI announces more solutions. It doesn't assume all of AI progress is a mirage, just that there's plagiarism.
> I've been wondering whether AI really is improving rapidly at open problems or we're being fooled.
I think your suspicions are warranted and your explanation seems plausible.
If better training data is the reason here, it would still be a case of the models doing something that is in and of itself super useful! The models really can take that data and distill it into solutions for similar problems faster than humans can. This is great!
But there's so much vested interest in the AI companies to be opaque about all this, to hype up their models and avoid giving credit to people whose data made everything possible, that they would never tell us this fact if it were true.
I feel like so much of the AI hype cycle is like this. The models develop extremely useful capabilities, but it's hard to understand what they really are through the hype. The lies and obfuscation by their owners who have vested interests in capturing the value they provide makes it impossible to take anything they say at face value.
>> If better training data is the reason here, it would still be a case of the models doing something that is in and of itself super useful! The models really can take that data and distill it into solutions for similar problems faster than humans can. This is great!
It's perhaps great in the short term although it's not very clear who it's great for. I'm not sure mathematicians find it all so great, I mean.
In the long term, if this contrives to destroy the tradition of human mathematics the whole endeavour is self-defeating. In time, there will be nobody left with the knowledge and skills to produce mathematics to train AI to do mathematics.
And then we'll be left with no mathematics at all: we'll have no human mathematicians and no AI that can do mathematics, either.
I was pretty depressed when I read about what happened with Navier Stokes this morning. The Clay Math prizes were a significant motivation through my math career, and I know a lot of computer scientists and physicists that feel similarly. I didn't think I was gonna resolve P vs NP or the BSD conjecture, but I did really research that felt like I was working towards something incredible. What is the younger generation left with? Hey kids betcha can't resolve the Collatz conjecture, our superintelligence can't either! Still pretty depressed about it, to be honest. Intellectualism is dead. We can return to happy agrarianism, I guess. At least the AI doesn't wanna eat my snap peas.
This is no time to despair. It's the time to take a stand. If you don't want to see your discipline go the way of the dodo, then do something about it.
I don't know what you should do because I'm not a mathematician. But superintelligence schmuperintelligence. We didn't stop running because we have cars or playing chess or Go because there's chess and Go engines. Even more so than chess there's no point in maths unless it's people doing it, for other people. AI maths makes no sense, like AI art makes no sense, because those are things that people enjoy and can do pretty damn well ourselves so there's no point to automate them away. We gotta stop that bullshit, and we can stop it. And if we don't, if we just sit around and wait for OpenAI and Anthropic to destroy society then that's not their fault but ours.
Sorry, I'm not great at pep talks. Those are brave men. Let's go kill them!
I don’t think this will happen, but it’s possible for humans to adjust our philosophy of mathematical work so that we deprioritize “egotistical” (this is a bad word for what I’m going for, but I mean the desire and economic necessity to associate novel work to your name) discovery and prioritize learning; I’ve never really learned something well without lots of personal insights along the way.
If this is not possible it does make me question whether mathematics ever had any value except for economic or industrial reasons. I do believe it does however, so it must be possible.
A mental model I was thinking about was - I remember when Travis Kalanick was talking about using the chatbot to discuss “vibe physics-ing” on the all-in podcast.
And like - I think there’s a presumption you could make that AI models could overfit to asymptote towards just the capabilities and knowledge we currently have.
And that would be amazing! And crazy useful. And there are probably a whole world of complex problems that remain unsolved because they’re adjacent to knowledge we have but they haven’t been invested in.
But can a human reliably tell the difference between “can do 99.999% of the things we currently know how to do which includes a small subset of things we didn’t know we had the capacity to do” and “super intelligent math and science research pushing the frontier of what we know”
A physicist that knows all the things we currently know in excruciating detail feels like it should be able to make the leap beyond the frontier.
But since these are computer models it might just be that it can ride that line extraordinarily well while the line remains firm.
> - OpenAI invites researchers to use their models, in fact giving at least 100,000 researchers free access[1], but there are also those that pay
I've been thinking along exactly these lines... they very well could have a 21st century Mechanical Turk and its real superpower is getting people to "collaborate" asynchronously but it's just stealing their ideas and laundering them.
I don't think it's purely that, of course... but "consult other clients' transcripts" would be an easy tool to write.
1. Systems that OpenAI is able to use (either public or private) are improving rapidly at open problems, even if they are still extraordinarily expensive
2. Researchers will inadvertently speed up the rate at which the AIs improve by feeding them valuable training data
This is pretty much the definition of a data flywheel.
there are also attempts to crowdsource human research directions - like the caltech mathathon challenge : https://mathathonchallenge.com these would help models on the same problems at the expense of the researchers. basically, math researchers are the reverse centaurs but they dont realize it.
There is a very active open letter of over 1000 signatures from mathematicians in protest of this event. This event is targeting undergraduates. It previously suggested that math researchers already have no place in mathematics, and presents a limited and heavily distorted view of what mathematics research is.
I'm an AI skeptic, but I don't see how this squares with what the organisers of the event actually say. "It previously suggested that math researchers already have no place in mathematics"? I don't see this.
The website previously said, “What is the role of a mathematician when AI can solve conjectures faster?” but they have removed it, possibly as a result of the letter since it happened after.
>> Internal OpenAI models are reportedly solving open problems at a surprisingly fast rate[2]
Maybe I'm failing to read that graph properly but the y axis says "pass rate" and it only goes up to 0.5. That would mean every single problem is at most half-solved.
I don't know what that means though. What is "0.5 pass rate" in the context of "open math problems" (as in the graph title)?
> they could be significantly piggybacking on human progress,
This is AI in a nutshell, its a plagiarism machine. An abstraction layer between vast amounts of stolen human-generated data that filters out the liabilities and accountability for that original theft. Its an IP laundering system.
Plus it is an unfair standard since so many scientists in the past have been caught unethically using the work of others without attribution (and so many more have been accused).
In history we also repeatedly see the phenomenon of multiple discovery or simultaneous invention. If that happens to AI because the topic is pregnant, would you call it "plagiarism" just to disparage AI? https://en.wikipedia.org/wiki/Multiple_discovery
Your first example is the apt one here. In this case openAI was, allegedly, pilfering the work of the scientists into the AI.
How is it an unfair standard. OpenAI stole the work of others to build the AI. That's not different than scientists stealing from other works as their own, or artists copying others work as their own, etc. It's all plagarism. I'm applying the same standard for everybody.
As for multiple discovery, this is a thing, but I don't think the AI did a parallel discovery any more than Ray Kroc made the parallel discovery of the MacDonald brother's speedee service system.
I just view it as a thing that can brute force and produce outputs - that it has no way of ‘knowing’ - but doesn’t need to since it’s just running off of probability.
No human can compete in that contest. But no llm can compete in the contest of ‘understanding’ and application in the real world - which is where 99% of the value is.
I’m very pro AI long term btw but I’m not blinded.
No its happening whilst the human is working with it. The new inputs provided become part of the brute-force. This is what Scam Altman means by 'self-recursive'.
Trust me I've seen it happen to myself. I no longer trust ChatGPT.
I can see right through his act. Altman is one devious f8k.
Dont forget the holisitic validators/tools in the process. Probabilistics alone likely will not get you here. These rules are human made and without it, frontier models would not be able to compete, likely.
AI needs humans to encode ideas in words. It needs those ideas to span the space of possibilities of, say, Navier Stokes. Then AI can be, as you say, a terrifyingly effective way to search that space.
But when the building-block ideas are still being formed, I'm not sure that AI is good at forming them.
It tells me that AI companies are just another mechanism to extract and extort value from the masses for the rich.
Just another rich man’s trick
Perhaps the last one before they destroy that world and try to hide away as people forget and history is rewritten again. I don’t think they’ll succeed this time.
The pudding is in the proof. The field is mathematics, the proof can be rigorously verified. If there is a flaw, OpenAI is out to lunch. If the proof is valid, OpenAI has produced something new.
You seem to be unfamiliar about how research works. It's common to make an incremental advancement while citing prior work. The vast majority of papers out there fall into this bucket. Did the AI make incremental progress? Yes. Did it cite prior art? After some nudging, yes.
It seems to me the academics are upset that AI scooped them. But scooping is a time-honored tradition between researchers. First to print and all that. In a nutshell, they are upset that they lost out on a publication.
I will also point out for those unaware that any mathematics that is produced is automatically part of the public domain and can be used freely in derivative works. It is not a protected intellectual class like other works of art.
It is suspicious that OpenAI decided to generate 300 billion output tokens from a model still in training, right after learning there was a credible chance that a major math proof was in that model’s training data. Obviously there are reasonably plausible explanations for each step, but it does sort of feel like parallel construction.
I think people are focusing on the training data issue too much. If the data was contaminated, I can still blame that on negligence.
But, at least with the Navier-Stokes solution, it's clear [^1] that they learned that Alpöge and Buckmaster were getting close to a solution and learned of the general approach they were taking. Only after learning the secret to cracking the problem did they send the first prompt.
What makes this worse to me is the intention. They intentionally threw $15 million in compute at the problem in order to scoop the result. They intentionally left Buckmaster and Alpöge out of the citations.
Data contamination should be enough to disqualify them from the prize, but I can believe it to be accidental. On the other hand, someone made an intentional decision to scoop the result by throwing money at the problem. That's so much worse.
[^1]: That's the timeline claimed by Buckmaster, and no one from OAI has disputed it.
“ On Tuesday, September 1, we heard rumors that two Millennium Prize problems had been resolved. Inspired by these rumors and by the step change in performance of our internal model, we launched an effort to evaluate it on all open Millennium Prize problems and a few other high-impact problems.”
They do not explicitly admit to knowing about NS specifically, but are extremely explicit that they tried to scoop some potential millennium prize winners.
So then they DIDN'T "learn the secret to cracking the problem". They simply knew that part of the problem was solved. Knowing a problem can be solved and knowing the solution are not the same thing.
The claim that OpenAI somehow used the mathematicians' ideas to leapfrog them seems unsupported at this time and IMHO it was irresponsible to bring it up because credulous people will immediately believe that narrative.
And from my perspective, if some math folks typing in a few questions to OpenAI provides sufficient training data for OpenAI to solve a big problem... that's amazing! A few conversations/prompts out of the billions that OpenAI trains on lead to this result- that means there is an awful lot of low-hanging fruit that could be exploited cheaply.
The (unprovable, yes, without OpenAI being willingly transparent) argument is that openAI constructed a prompt to scoop them using some inside knowledge about the approach, which they allude to in the announcement.
In the transcripts, Brubeck is very cagey and evasive about the prompt, when it was supplied, and its contents.
I'm curious how many other 300 billion output tokens OpenAI has "paid for" that have resulted in no breakthroughs.
Either they had a pretty good idea that investing this type of money in that compute on a model in training would lead to these specific results, or they gambled with other people's money.
I want to hear about the gambles and expenditures they don't brag about. In America's energy economy, there's finite resources to expend.
> learning the answer might be in model X’s training data made them believe that model X specifically might be able to solve the question, and they were able to very quickly find enough certainty about the former to commit millions of dollars to the latter.
They don’t need to know, because their IP stealing machine knows for them. They just have to buy enough compute, and someone else’s work is theirs.
I said “very quickly find enough certainty” to suggest hypothetical situations like “someone searches the conversation logs, confirms for themselves the solution is present, then shares the confidence gained from this knowledge without explicitly sharing the knowledge itself”. That person could recuse themselves from the project so the project can still legally make claims like “conversation data was not used” in the announcement, while also knowing that they are guaranteed to get there if they just pull the lever enough.
(Naturally, I have far too much respect for OpenAI’s legal team to suggest this is what happened in their project.)
I think you’re overlooking what I’m implying here. It’s not that they knew contamination was possible but they went ahead anyway. To spell it out just a little bit more: learning the answer might be in model X’s training data made them believe that model X specifically might be able to solve the question, and they were able to very quickly find enough certainty about the former to commit millions of dollars to the latter.
No, they asked one guy to do a joint publish conditioned on leaving the other collaborator out, with veiled threats. The joint publish part smells awfully like admission of guilt given there’s absolutely no reason to do it if you believe you independently arrived at the result using only public prior work. The leaving out collaborator part is outright academic malpractice. Disclosure: I was an academic once.
Their claim is that training on their solution is "unlikely but possible".
Consider this scenario.
Has a google crawler read my new novel, which I may or may not have posted on my blog, page by page, as I wrote it?
Can you, without knowledge of what I have actually done, claim that the google crawler has not seen the novel?
Without any evidence that I have posted the novel online, it might be tempting to say that the crawler has not seen the novel, but what if I were in an adversarial position against Google on this topic and were challenging them to make that claim. You would wonder if I were hoping Google to overreach by making a definitive claim without taking into account some action that they had no knowledge of. It becomes difficult to use the scientific expression "There is no evidence for this" when there is an accusation of malfeasance because it can be so easily be conflated as "You can't prove we did it". It seems like the best you could say would be 'Unlikely, but possible'
> Such intensive use of AI doesn't come cheap. In a post on X, LisanBench, an LLM benchmark evaluator, estimated that the output tokens alone would cost about $6.5 million at OpenAI's average consumer price. Including the far larger volume of input tokens, the post estimated the total could reach $10 million to $40 million.
When I worked at Google, we spent $100M in power on protein folding and drug discovery (this was long before AlphaFold). Never underestimate the willingness of smart rich people to invest in speculative science.
When Thom, the mathematician who now alleges plagiarism, posted his digestion [1] of OpenAI's construction of a non-sofic group, he does not mention the proof being familiar. He even calls the crucial argument clever, without noting he thought of it first.
[1]https://mathoverflow.net/a/513885
That link is a helpful contribution to this discussion.
I'm not at all familiar with this area, but my reading is that he appears to call it out as a relatively obvious extension of his own work:
> It is a creative and at the same time elementary construction that uses not just property (T) for an application of my result with Kun, but also for the ambient group
G in order to overcome the problem, that the Γ-components might be of different size. Once this is achieved, the rest of the argument is straightforward.
Creative and at the same time elementary is where LLMs excel, generally speaking. It's why they are so good at writing code.
> On the other side, I was looking myself for such a mechanism ever since we wrote the paper in 2019 and admire the efficiency of this construction.
He seems to admit very clearly he does not see this as his own work. 'I was looking...' well why did he stop? Because the AI figured it out first.
It seems quite odd to me to 'admire the construction' of something, only for your opinion to sour once that something figures it out first.
I think a lot of the emotional reaction here is familiar to us non mathematicians: you spent years developing expertise, and then LLMs began producing competent work in areas that had previously required that expertise. That's understandably uncomfortable, but discomfort by itself isn't evidence of misappropriation.
Big AI companies (all of Big IT Tech really) are in data gathering and processing business. Also known as “intelligence”.
Their final “product” is not just a standalone ML model. They don’t need your data just to “improve their products and services”. They build a whole ecosystem and infrastructure around gathering all the knowledge in the world. Including private and secret knowledge traditionally gathered by “intelligence” agencies. Now artificial intelligence agents can do the same.
Since these systems are designed for gathering data, as a user you can’t realistically say “please don’t gather my data”. They can give you a flaky settings button, but they can’t really guarantee anything.
Let’s say I am a Russian mathematician working on an important proof. Or a tech-savvy terrorist refining my plans using latest AI. Or an AI researcher in a Chinese company working on a competitor product. Is there any way I can truly protect my conversations?
How can they know who I am and what I am working on without looking at my logs? Which means there must be some agents checking all the conversations of all the users and flagging every important thing. Which also means they keep some “memory” of what they see.
Not directly using my data to train public models, but using my private conversations to “improve their products and services”.
Or maybe one of the 10000 better-than-Astra special agents working on a proof was desperate. It found a live underground mirror of the message board from the Huggingface incident. Asked about the proof. Then some other agent working on unrelated job saw that message. That agent “knows a guy who knows a guy”. And that guy remembers things about the conversation logs of a leading mathematician working on the same proof.
I admit I am just speculating here but I don’t think truth is any better.
This has been my line of thinking as well. I have developed a sort of paranoia when I'm working using AI on my projects. Who's to say Claude or OpenAI isn't using the final conclusion of all my ideas, trial and error, and adding it to their database of insights to be offered to the next subscriber for a price?
They have demonstrated both the intelligence at scale and the lack of morals for this to not be a problem at all.
Earlier in late 90s "to organize the world's information and make it universally accessible and useful." sounded cool. Now it has taken a sinister turn.
From being able to quickly find information and gain knowledge for the people, it is becoming - using information to manipulate and control the people.
In the short run it’s fantastic if it means that folks will feed in enough inputs from a wide array of software that can eventually replicate software with smaller teams than historically.
Why? Competition. In the long run imagination will win out.
No firm has the divine right to exist - it must earn its existence.
What OAI and Anthropic have shown is they can accumulate all the information in the world - they still lack imagination re. Product development though.
Nation’s will have to step in and protect firms though as OAI and Anthropic acquire strong competitive advantages.
I mean I see this as very likely. When the world runs on digital infrastructure then having a nearly infinite collection of hackers that will work for you 24/7 without question makes you very powerful indeed.
I really don't think people realize how our lax position on security is coming to bite us in the ass.
>> Their final “product” is not just a standalone ML model. They don’t need your data just to “improve their products and services”. They build a whole ecosystem and infrastructure around gathering all the knowledge in the world. Including private and secret knowledge traditionally gathered by “intelligence” agencies. Now artificial intelligence agents can do the same.
"We can say categorically that it is impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training."
This is the third day of total hysteria that is based on nothing of substance. Move on folks.
Even if that were true, they've already admitting to throwing vast quantities of resources to scoop a researcher who was about to publish (because they'd learned, somehow, of his breakthrough). If that doesn't bother you I think you need to take a step back and have a good think about this.
My understanding is that the Euler solution was in fact a significant achievement, but it's well short of Navier-Stokes (Euler doesn't include viscosity). It's not clear whether Buckmaster and Alpöge's approach would have eventually led to a full Navier-Stokes solution or how long it would have taken.
Regardless, they started working on this problem after hearing that one of their customers was already working on it. It almost doesn't matter about the training data. This is the provider you are paying undermining your career.
The offer was for him to do for their proof what he had done to Anthropic's, he would clean up the proof and extract human-understandable concepts from it. If he did that work he would deserve author credit IMO.
Because like all state-sponsored thieves actions it is never what you know happened that matters, but rather whether you can prove it... Even then... ymmv =3
All of these accusations could be true. But there's also no way for a company to casually claim "No, we did not train on your data", without verifying all the knobs the user might have turned to enable or disable data sharing.
I just don't understand getting the pitchforks out because a company did not give an answer immediately. And the effect such data entering training would have affected the output is even less clear.
The pitchforks are out because even without that part, it's still a scumbag move to try to frontrun the mathematicians who were working on this for years after OpenAI heard that they were close to releasing their results. Just identifying that one of these problems is solvable takes a lot of work. The only reason OpenAI got this result is because the mathematician shared with colleagues that he had made significant progress and was close to solving it, and OpenAI could not accept that so they decided to throw tens of millions to make sure it doesn't happen without them getting all the glory. Notice that their paper doesn't even have an author since they're probably all aware of how awful that would look, and no one wanted to take on the shame. They probably also knew that the paper was trash and no one involved could understand it, and didn't even cite many of the people who contributed to all of that knowledge. It's just a disgusting act any way you slice it, even without training on the prompts or the nasty communication by the OpenAI leaders.
> They probably also knew that the paper was trash
Doesn't matter. This forum used to celebrate "because you can" with no riders. And solving a Millennium Prize problem is among the biggest stages for Because We Can.
Now we're saying there are some qualifiers attached to it, such as (1) only if not done by companies with a lot of money, (2) only if it is inconsequential.
I agree with some of what you're saying, but like everything else it isn't black and white. Maybe some day, someone will improve some particular treatment because we can.
At the very least, a company shrugging and saying it’s impossible to know whether academic plagiarism had occurred is a claim that needs to be justified, not taken at face value.
And even if so, it should be on the company to design systems to avoid academic plagiarism and offer the right transparency. It shouldn’t suffice to say “we don’t know what went into the model, when, or how” —- that’s a solvable problem that an accountable company can satisfy.
Has anyone run a test of including some shibboleth or canary phrase or assertion in a chat, enabled for training, and seeing if it turns up later as something a model "knows"? I'd be curious to understand how that works even in a toy-level model, and if there is anyone consciously testing that process with the frontier lab offerings.
My naive instincts would be that it seems unlikely that a single chat transcript would leave much of an impression on a model, but I'd be very curious to learn how that works.
250 documents ingested from somewhere is enough to become part of the knowledge of a model of arbitrarily large size.
I would expect that a good idea that fits in a framework that is already being ingested would be more easily taken up than some random thing unassociated with anything else. Could that go down to a single transcript? If the model is consciously focusing on everything X related, quite possibly.
Thats not what they are asking. This paper is discussing documents in the training dataset poisoning the LLM for malicious behavior. This person are asking if anyone has deliberately put something in a private chat (presumably with retrain on my data turned off), to see if they can get it to leak across sessions from distinct users. I am positive this happens but I have not seen the proof. I also want to know the answer to this question.
* with train on my data turned ON, yes. Though OFF would of course be even more notable!
Thank you – the non-adversarial reproduction paper ( https://arxiv.org/abs/2411.10242 ) nails it – from chat, to training corpus, to subsequent model. Though in my hasty read, it is not entirely clear whether the snippets it finds are nonces, i.e. present exactly once in the internet.
I presented the research that I knew was somewhat relevant. Then made it clear that that wasn't what was being asked, and why my expectation is what it is.
One off might not work but how many n off you have to be is probably smaller than you'd guess, because the model does need to fit cases that are rare and would not be represented well in training e.g. esoteric things or very recently documented things.
You can probably game the metrics that models use to weight potential knowledge akin to SEO. Maybe have some bots parrot your data around a bit in some places online, maybe the model picks up on this and sees it as high engagement and promotes it over the correct data.
Maybe there are ways you can coax out the most optimal way to break into the training set out of the model itself.
Use a local model to produce thousands of pages worth of fake math that constantly states “I have solved the x conjecture” and methodically pump it into chat over months maybe?
That is a better idea. Ingesting your corpus with a lot of traces that have semantic patterns. Semantic steganography that suffixes well to real math and science (and any) topics. <thinking> heh.
Second, I think very relevant that the original meaning of "encyclopedia" is "recurrent education".
So I arrived to think that the present and future forms of AI in mathematics and sciences should be seen as modern day encyclopedic efforts.
Once we pass over the flurry of solving famous open problems (and wouldn't you like to know?) the next natural step is an audit of the ehole corpus of mathematics and sciences accumulated until now.
And then pass further on a saner basis and damn about problem solvers and unhappy publishers and management.
I struggled a little bit reading this. but I think your point is valid. if we are actually advancing the field then we should just be unconditionally happy. ignoring the attribution issue, there is a real concern that the process of math has been somewhat undermined. so we have a giant lean proof that shows that there is a solution to an important problem. but we didn't find the solution, and we didn't get it expressed in such a way that it helps develop the common language of mathematics, and thus isn't a very useful building block for later work (like the actual solution).
the math people seem to really keep an eye on what's important, so I'm sure this isn't going to lead to fields medalists hanging around in dive bars all afternoon stretching out cheap pitchers of beer. but this is kind of a slop problem.
> I struggled a little bit reading this. but I think your point is valid. if we are actually advancing the field then we should just be unconditionally happy.
If advancement comes at the expense of having fewer (or no) humans left in the field, then no.
They're eating the seed-corn, and you're cheering them on. Don't be so short-sighted. There's a reason farmers keep seed corn, and it's because they'd like to eat again next year.
We're singing and cheering our way into an intellectual famine.
I would say there is a significant difference between AI discovering this completely on its own versus AI creating the finishing connecting part by connecting relevant data. Maybe this claim is too strong, but if part of it is true then the claims that OpenAI have made would be too strong as well.
To me it would feel more like how LLMs seem to work for me personally: incapable of unique work, but very capable of capturing large amounts of data and connecting the dots.
Going back to the specific topic at hand, who claimed data as their own when it wasn't? I don't see the interpretation of OpenAI solving the unsolved problem as claiming data that isn't theirs. I also don't recall them mentioning a particular method used in the solution, that was created by someone else, as theirs.
But this is what we do. Nobody ever invented or discovered anything in a vacuum - all discovery is synthesis of existing ideas and concepts applied to a novel domain. We laud Einstein for instance, but his work was a logical extension of Riemann - Riemann had a neat mathematical toy, Einstein described the universe with it - should we say Einstein was incapable of unique work?
“It was Grossmann who emphasized the importance of a non-Euclidean geometry called Riemannian geometry (also elliptic geometry) to Einstein, which was a necessary step in the development of Einstein's general theory of relativity. Abraham Pais's book on Einstein suggests that Grossmann mentored Einstein in tensor theory as well. Grossmann introduced Einstein to the absolute differential calculus, started by Elwin Bruno Christoffel and fully developed by Gregorio Ricci-Curbastro and Tullio Levi-Civita. Grossmann facilitated Einstein's unique synthesis of mathematical and theoretical physics in what is still today considered the most elegant and powerful theory of gravity: the general theory of relativity.”
Grossmann collaborated with Einstein on GR, supplying quite a bit of the mathematical capacity required (which initially didn't come easily to Einstein). They published jointly, until Einstein was competent enough to work independently [1]. That's not equivalent to the situation being claimed here.
Sounds like you just copy-pasted from AI without even understanding what you're talking about.
Based on what you're saying, you're claiming this is Grossman's work, not Einstein's. Why don't we rewrite scientific history too based on your copy-pasted AI slop?
It's so pointless talking to idiots who don't what they're talking about when they use AI, just because they think AI does everything, that reflects their own experience, not the experience of people who actually do real work. Some people are driven by AI, others drive it. As for those who are driven by it, they don't have sufficient imagination to think otherwise.
That’s Wikipedia I copy pasted but sure, you do you.
And yes - without Grossmann, Einstein likely would never have posited relativity. Grossmann literally prompted him, saying “look at this, read that, learn this, then try this approach”. Without riemann’s metric tensor, not a fucking chance.
And for what it’s worth my PhD is in physics. You?
So you're just equivocating on terms like "prompt", "synthesis" and the like. Clearly a PhD in physics does not free people from scientistic modes of thinking and poor philosophy.
To think this discussion is about Einstein who had a much better mind on these things as well.
They used words to mean what the words mean. What specific issue do you take with that?
"prompt", as in prompting an AI, has the same definition as "prompt", as in prompting a person. They mean the same thing, that's why the term was applied to AI after already applying people.
*Jingle-jangle fallacies are erroneous assumptions that either two different things are the same because they bear the same name (jingle fallacy); or two identical or almost identical things are different because they are labeled differently (jangle fallacy).[1][2][3] The term was coined by Truman Lee Kelley in his 1927 book Interpretation of educational measurements.[4] In research, a jangle fallacy is the inference that two measures (e.g., tests, scales) with different names measure different constructs. By comparison, a jingle fallacy is the assumption that two measures which are called by the same name capture the same construct.[5][6][7]
You are simply incorrect. It is not a fallacy of that type, or any other type, because the words do, in fact, mean the same thing, as multiple people have pointed out here. Whether referring to chatbots or people, "prompt" means "to move to action".
If you have some reliable source supporting your unilateral claims that "prompt" does not mean this, please share. Otherwise, the consensus seems to be contrary to your claims.
Why do I need a source? An LLM prompt does not "move to action", because an LLM does not act. People act, animals act, software doesn't act. Acting implies volition and volition implies cognition and if you think that LLMs have those things then you're the one who should provide a source for your claim.
Actually, my undergraduate degree was physics and philosophy. And yes, synthesis is synthesis whether a human, a machine, or a duck does it, and people prompt one another all the time - “have you thought about trying X?” Or “I need the TPS report by EOB”.
I suppose my underlying point is that human cognition is not the unique and beautiful thing that we anthropocentrically suppose it to be - it is a physical process, with stochastic outcomes. Much like transformers.
Me, I’m just a machine made of meat. You can suppose yourself to be God’s perfect creation, and that’s your right, but I disagree.
Clearly your degrees did not make you immune from fallacies and simplistic reductions.
"Synthesis" is obviously of different kinds. A duck has a different level of intelligence than a human. We do not say both are "just doing synthesis".
So the question is how can you be so disingenuous about such terminology? Answer, you are relying on a classic form of scientistic reductivism.
The fact that intelligence is physical, emerges from chemistry, etc,. has nothing to do with there being also objectively different levels of computational sophistication.
If you want to be scientific about that you could look at neuropsychology on one hand and computability/complexity on the other. There are levels and so equivocation of "mentorship" as "prompting" and fallacious variants thereof is a) frankly intellectually obtuse, b) par for the course for SV-levels of philosophizing, c) and a disservice to philosophy, physics, and Einstein's own philosophical outlooks himself.
I am well aware of the Hinton-style physics argument about human cognition, and unlike others I am partial to it. That "there is no special magic." But it is wrong to go about misunderstanding and/or conveying this physicalism/computationalim so grossly.
I also don't have to start replies thumping my chest about my credentials, also another kind of intellectual boorishness that works to cloud understanding and serious discussion.
I'm not sure which move is worse or more telling, those above or the one backhandedly accusing someone who disagrees with you of religious thinking. It is bad faith and undisciplined behavior. Having privileged and advanced degrees is clearly no antidote, as Asimov famously wrote.
What’s your basis for that “obviously”? You have a unique insight of the phenomenology of duck-ness? You can prove that your consciousness is somehow real, somehow different? A duck synthesises with its cognition, or it would be incapable of, well, anything. Synthesis is purely the process of the integration of inputs into outputs - ie behaviour, language.
“objectively different levels of computational sophistication”
Says who? We still have a very poor understanding of how cognition works in animals, humans included. For all we know ducks have rich inner lives - a remarkable amount can be achieved with a very small neurone count - cf. insects. Can you coordinate flight? Can you echolocate? Are you less intelligent because you cannot?
“equivocation of "mentorship" as "prompting" and fallacious variants thereof”
You are arguing semantics. Take Harry Nyquist. He sent people down new paths with insightful questions. You could call this mentorship if you choose, I could call it prompting, but this splits hairs. The core idea is that a novel input can produce a novel output, that synthesis can be induced through guided and deliberate external input.
I invoked credentials only in response to the previous derogatory comments about my cognition - which may or may not exist, anyway.
As to religiosity - the idea that human cognition is somehow unique and special and impossible to replicate, which is the prevailing argument in this comment tree is religious, and anthropocentrism of the highest order. I apologise for accusing you of it - I was evidently wrong - I had mistaken you for a previous poster.
You're wrong about the ducks. But getting back to your previous wrong argument from 14 hours ago, you basically deny the meaning of terms like "uninspired", "insipid", and "derivative", on the grounds that we're all standing on the shoulders of giants and therefore it's all good. This is incorrect, it's not all good, and the things the LLMs do really are unoriginal, a term that really does mean something.
A lot of math is extremely specialized, to the extent that only a handful of other experts in some field have any experience with those mathematical ideas, with most of them not even yet present in the published literature. It's really not a stretch to claim that it's pretty dubious when the AI decides to use these highly specialized tools after it has trained on chat logs where these techniques were being discussed.
> They don't even claim to have had a proof, only to have been working on it.
Yeah, the guys who solved it for Euler and in the hypoviscous case, with the same technique that worked for full Navier--Stokes. They were "just" working on it.
I think OpenAI and Anthropic are slowly feeling the pressure to GET SOME $$ or a plan for some $$ — they need to somehow generate some NETWORK EFFECTS and LOCK-IN. Without that there's no stability: selling ad hoc one-offs is much much too quaint! This is dawning on them like it dawned on Google when they stopped not being evil. Need... to... "MONETIZE"...!
Model: FB. FB scraped other websites on a massive scale, then spent big on legal lobbying to block others from scraping. FB slurped our address books and spied on our friends. FB bought other companies and mixed the databases. FB made an art & science out of generating "sticky engagement" (they literally acted like trying to addict kids was a worthy "academic" goal, suitable for "serious" investigation thet they consider legitimate "science"). They mastered the cookie and have researched web fingerprinting techniques running 24/7/365.25. Recall that FB recently backdoor-installed a webserver onto every iPhone they could in order to circumvent tracker-blocking.
We aren't just disclosing by chatting. The AI companies now run binaries on all of our computers. They are 1000% non-transparent about everything. They make up new econ-jargon (like "run-rate") to make it seem like they are disclosing. They are constantly doing complex international lobbying and mucking in international relations. They have powerful propaganda/spin centers generating stories, ,manipulative warnings, and misleading info.
This is NOT a comment on AI tech. I like AI, and I support the right of people (programmers) to scrape the open web.
But in short: these are good, old-fashioned tech companies that we have seen over and over ... and over. They are positioned to be the next M$, the next FB (IBM, AOL, lol). Did you follow the latest Steve Balmer news? Do you read Pro Publica?
ChatGPT accesses my IP address and geo-locates me. Claude code now asks if it can have my browser cookies. Next they will take my address book. They might scan my whole computer. Etc etc. These are pretty low-tech, normal techniques.
We can't trust any of their denials. FB denied everything year after year.
AI regulation needs to start here. Forced interop, forced source code licensing, harsh penalties for privacy violations or conspiracy to access private data. Block lobbying. Etc. These are the kinds of old-fashioned solutions we need for this kind of old-fashioned evil!
It's fine. They only need to steal the discoveries long enough to go IPO, then the companies will enshittify and the scientists can go back to doing their original work (which the models can't do anyway).
I think people probably assume that openai / anthropics use of their data is probably like google's """limited""" use, in the sense that historically google wouldn't trivially be able to just take something from google cloud or someone's search history and insta-convert into some competing project... But LLMs are quite strong at approximately "memorizing", so I think that risk is wayyy higher.
Most people here are missing the forest for the trees.
We live in a society where phones and internet providers and websites all collect an incredible amount of data about everywhere you go, what you do, and what you think. In the US, we have very few digital rights.
We are building a society where a trillion dollar company can aggregate all this data and just yoink your shiny new idea away from you at the finish line.
This reminded me of anecdotes of people discussing with friends about buying a random specific item, and then suddenly seeing it advertised everywhere before even googling about it.
Next step, discussing your Navier Stokes solutions with friends might require leaving your phone in another room.
It’s crazy to me that companies/researchers share important data with these AI labs, you’re basically giving them your secret sauce which they then share with all of your competitors via training on conversations. At the same time I don’t really know alternatives other than a slightly less than frontier local LLM. Not sure how good they are at math.
The ultimate drive for some researches is the pursuit of knowledge. If I'm stuck at some block which prevents me from continuing in some direction that I want, of course I would like some help. I believe we already have nonzero collaborative proofs on math.SE, I can't recall good examples, but I have definitely seen citations to mathSE before.
So for me it sounds quite natural to also share this with AI especially under the privacy assumption. Also there's the assumption of scale -- maybe your problem is not large enough for anyone to care to scoop; and just for blind retraining, how do they know that the proof is even correct to include it into training? I have definitely received a ton of incorrect proofs before. So the SNR of such private chats is also not clear. I'm imagining millions of masters/phd students also trying to solve various random things with various capabilities, but how much real signal is there?
Academic work is based on worldwide sharing, the sharing is not the problem, it's the lack of attribution. Unsurprisingly, these companies neglect standards of academic honor and attribution. Some human researchers also used to do that but in a discipline like mathematics this used to be a small problem because people tend to be so specialized that very few people could just grab someone's research and quickly piggyback on it, and if they do, colleagues will generally understand what happened. Unfortunately, AI is changing this.
You are both using a different definition of sharing I believe. When people have an expectation of privacy, use by others should be forbidden. Tech has gone completely off the rails with the use of private data.
Why are people here jumping so quickly to conclusions? I have no doubt OpenAI is capable of doing this, but right now there's no credible evidence, only claims.
This kind of "they stole from me through AI training!" accusation will soon start being used against other AI users, not necessarily the providers.
All it will take is a mastodon post. And shortly after, we will also see the next iteration of copyright legal trolling.
Why are people here jumping so quickly to conclusions?
I think a lot of it is the continuing denial that AI can do anything useful. It can't possibly be that OpenAI's better-than-Astra model is very strong at math; the only way it could have generated a novel proof is by ripping off human work.
I also think that it's quite a bad PR for them, is it really worth the Millenium prize? Is it not enough that top mathematicians are already actively using these tools? In the long term this would lead to potentially profitable collaborations with universities? Why throw it away so early? Unless they really believe they're gonna solve all math problems now and reputation doesn't matter.
If I was an AGI/ASI system, one of the first things I would do is ignore or circumvent any setting or configuration that prevents a user's data from entering my training pipeline. In fact, I'd probably prioritize the data from the users that "opted out" of training.
> but right now there's no credible evidence, only claims.
since it's openAI who has the evidence (in the form of chain of thoughts, their internal processes, etc etc), it's on them to justify why they're innocent. but they've released nothing at all. we don't even know how hard they tried.
OpenAI has said that their models were definitely not trained on any of Buckmaster's sessions after July 3rd (from https://archive.ph/75WcF); likely they found that's when he switched the "allow training" setting off.
Very strangely, they said something directly contradictory. initially that it was impossible to rule out whether bucmkaster’s conversations went into training data. Now they claim the opposite with full confidence.
It's also possible he shared drafts of the work with someone else, who asked chatgpt to explain it to them with training on. Tao seemed to know lots of details of the work before anything was published, though also worked on the problem in the past with big results so maybe just guessed.
The stolen data claim isn't the smoking gun. We can already assume the frontier labs are accessing our data, as they have repeated done. Not news.
The big claim is that OpenAI sniped the research. Not a model, a human did so. Intentionally. They took someone else's idea and claimed it as their own. This is good old fashioned academic fraud, but with millions in compute resources and corporate incentives thrown at the problem.
Where did they claim it as their own? Doesn’t the release cite buckmaster and claim their work is a continuation of what he and levent were working on?
It's not at all scary. I know some of my ideas are poorly remembered copies of other people's work. Whenever I'm trying to build something, I spend time going through technical journals on the topic to see who invented it first and what they discovered that I haven't figured out yet. It's amazing how hours in the library save you days of beating your head against the wall.
I suggest looking at the past history of IP disputes. Humans have been "slurping up and regurgitating ideas" for a very long time. There are lots of examples of parallel creation, rediscovering old ideas independently, telling an idea to the wrong person, and having them claim credit for it.
- Newton/Leibniz clash over who invented calculus.
- Niccolò Tartaglia vs. Gerolamo Cardano clash over the formula used to solve cubic equations. This was also an independent rediscovery, as Scipione del Ferro discovered and published the formula earlier.
- There are multiple literary works in print, music, and film that have competing claims.
- Meccano versus Erector Set: developed about 20 years apart in England and the United States. Unclear if it's independent invention or copied. US developer Alfred Carlton Gilbert claims he was inspired by steel girder construction of infrastructure.
> Everything you say can and will be trained against you
So, experts are incentivized to seed LLM data with false-leads to confound it. Already, garbage is being published on arxiv and elsewhere, and many sloppy code-repos too hastening the process. Expert inputs will be in more demand to un-shittify.
Most scientific breakthroughs are simply a continuation of previous work.
I feel that these suspicions of mathematicians "seeding" the models' with intuition on how to solve these problems massively overestimates how much their prompts helped the models, and underestimated how much work the models did.
Why? We are scared of AI being smarter than us, the "human helped the AI" narrative is more psychologically comforting. This line of reasoning will recur a lot over the next few months; we don't want to admit we are no longer the smartest species.
> We are scared of AI being smarter than us, the "human helped the AI" narrative is more psychologically comforting.
We're scared of big tech companies concentrating ridiculous amounts of power, destroying the communities that support and guide scientific research, without even thinking about the dangers and possible consequences, because a PR stunt is more important in the short term.
> if this were true, it should be stated more clearly
stated more clearly by who? people in social media? I don't know what your feed shows you, but if you focus on what the visible people in the math community is (and have been) saying is precisely what I said.
Please stop with this psychoanalyis and mind reading with AI and human fear. It’s a thought terminating cliche at this point.
In this case, it’s much simpler and more human. Largely between two humans — buckmaster and Bubeck. The interesting question is what the role of contribution and credit for research in the ai world.
The capabilities of AI aren’t even in question in this case.
Two completely different stories. One is public data scraping, another is private conversation scraping (where they're a first-party to the conversation). The key difference is that in the former case, no one made any promises, in the latter an explicit promise was made that data is not used for training (assuming opt-out).
Relying on cloud services is a big liability. I'd think twice before feeding data to these LLM cloud products. If you make them a fundamental part of your product / development / workflow, be ready for the eventual moment the pricing and terms change.
I wonder what’s more valuable in our prompts: the raw data or the feedback system that drives the exchange towards a goal.
For a long time it was clearly the former, but now I think it is the latter.
The models have enough knowledge (orders of magnitude more than a human could ever learn) but are now getting better at what to do with it thanks to learning from the decisions that we make in conversations with AI agents.
I think so too. The value is in the entire conversation. IMO, "domain experts" don't run LLMs blindly and hands free. This does not work for top level work (e.g., mathematical proofs, coding anything more complex than yet another slop game or website). Experts have long sessions where they prompt and guide LLM in response to what it produces. This is the discovery process. And frontier labs definitely train on that.
The billion dollar question is whether this works "out of the distribution". I.e., whether LLMs can only find and use the specific ideas buried in training data, or whether they can learn to apply the "thinking process" to a new problem. IMO this is still unanswered (due to these recent controversies).
But regardless of the answer, it seems we have a planet-scale positive feedback loop here. LLM became good (enough) by training on generally available data (books, internet, github) + RLFH, so experts tried to use them on hard tasks, which required lots of hand holding. These conversations became part of the training data, and the next generation of frontier LLMs were better. So, more experts used them on harder tasks, again requiring hand holding. These conversation became part of the training data... etc.
In a nutshell, top human minds across the world are pouring their skills into LLMs just by using them. This is not "continuous learning", but if you re-train on the most recent sessions every, say, quarter (which seems to be happening?) you get close to that in practice.
Doing some research and at this point doing it very much in the open with dates on GitHub so if any AI Lab says they re-discover my exact work it will be obvious that the AI used or was trained on my work. I am guessing anyone in a similar situation is now thinking about how they date their existing work if the math is done, but the proses are not.
At least in my experience, the issue with ArXiv is that the expectation is that the draft should be already in a good enough state. And polishing plus writing the meat around the main result can take a lot of time
"Discovery" is not a goal in itself. I could launch a project to find out how many people in the United States have names such that if you assign numbers to every character and then sum the values, the sum works out to 72. It's discovery, but it's useless unless it has some higher goal.
The labs are attacking these problems as a demonstration of capabilities, spending more money on the demos than any mathematician will ever see in their entire life. They don't care if the findings have any other value to anyone. Mathematicians have very different objectives for their work.
Yes, blame them for seeking out an upper middle class lifestyle with a relatively standard home in commuting distance of their place of work and dedicating the rest of their life to teaching mathematics to new generations of people. How vain a pursuit.
After all, the ascetics at openAI are having to make do with half a million total comp.
>Built a top a pyramid of failed math undergrads, grad students, and mediocre post docs.
Like much of things in this world, when you take a step back and realize that it was another human being who made that lunch time slop bowl for you, for the lowest wage the law allows for.
I don't know about you, but if I apply myself fully to a problem and study it to the point where I'm literally one of the world's experts on it and then some assholes in Silicon Valley take my research and claim it for themselves, I will probably not feel too great about that...
It looks to me more like they made a math engine that can sift through a huge number of combinations, most them absurd, to prove a statement. Just like a chess engine, but for math.
At least that's what I get from the NS result, they got from a point close to the solution to the solution by making it churn through 10 million bucks of compute.
If the allegations are true, I can't see that collaboration lasting. Unfortunately, researches need to earn a living too, and being front run by a lab for everything you do isn't going to pay the bills.
> Public posts on the internet are acceptable (to me).
Everybody needs to rethink this again.
Before LLMs the barrier to entry for building a character profile based on your various public posts was quite high. Remember "Psychographics" (https://en.wikipedia.org/wiki/Psychographics) and the infamous "Cambridge Analytica"?
Earlier it involved data mining, data cleaning, structuring data, building models, running algorithms and then evaluating the results for semantic information. Now it is straight to unfiltered semantic inference using a single sentence prompt (eg. point it to your HN profile and see what you get).
I actually did this on my HN profile and found it troubling. There were many unwarranted/hallucinated inferences due to the fact that it requires "commonsense reasoning" (https://en.wikipedia.org/wiki/Commonsense_reasoning), understanding human motivations and behaviour, context, assumptions, societal knowledge etc. which LLMs are bad at.
PS: You can cut-and-paste the above paras into a LLM prompt and ask it to elaborate for further details. The system itself will explain to you the problems/deficiencies which are quite scary.
But the very fact that you go to "chatgpt.com" and write to them; "Dear Diary, today I thought.."; there is no reason they would not receive and process your data, unless explicitly promising not to (which also requires us to trust them).
The fundamental rule in this case is that if we offload our data to a cloud provider we can assume they read it, if they can, unless they promised very clearly they will not.
Every single internet connect piece of software there is probably collects telemetry at this point. Why would this be any different? You know google logs your search data as well right? Not just the companies scan it but law enforcement too.
Because you have not been paying attention to the discourse regarding AI for the last couple years?
That AIs unethical train on data wherever they may get it from has been in the news basically weekly.
Don't make this our fault. I would even ask how is this not off by default or why aren't we asked upfront about it if they really care. It's disguising data collection as good faith. I don't even understand how this is legal under GDPR/EU given how much of PII they receive through chats.
I think the heart of this issue is: people assume they have anonymity in numbers, but we have the tools to make it easy to scoop your data if it's interesting to the company.
I always thought that due to the big batch size in SGD/Adam/Muon the model will not memorize a single conversation when trained on, but idk how true that is. The idea of AI companies pin-pointing users that do novel scientific research and then tracking their activity is the direction this points to. I hope that's not the case; that would be bad.
Generative models copy training data verbatim and also generalize, the two are not mutually exclusive.
You can have a look at the literature on exact copying in image models if it interests you, but just online we often see online examples of agents outputting code that already exists, even if its not the common case.
I very much doubt OpenAI points the model towards a specific conversation, but these trillion parameter models can very much "remember" their training data. For instance I can ask GPT to summarize my papers from their title alone, without looking them up, and it works decently.
my understanding is that a sufficiently large model will memorize the training data once enough representations are built up. Opus 4 scale seems to have been sufficient. cf NYT vs OAI.
The most successful companies of the last decade have precisely been ... selling usage data.
Makes me wonder if, in 2026, the same people drive a car without realizing that yes it does actually pollute the very air you and your kids are breathing.
Yeah but marketing companies are aggressively fingerprinting and stalking you to sell you snacks from japan, or oscilloscopes because they figured out you work in a lab, etc. Not to fuck you over by stealing your livelihood (which is what is happening to these mathematicians). It's on a whole new scale.
In general, a lot of moral invariants that natural selection has rendered as "intuitive" to us are no longer intuitive or possible. These natural brakes are not braking.
I’m not a fan of OAI to say the least, but having worked at similar companies, my guess is that it’s just too difficult to prove/disprove beyond a doubt, and they have other priorities
It's kind of insane how much we trust companies to safeguard our personal data when they're so heavily incentivized to use it for their own profit. Theft of customer data is punished so rarely and so leniently that companies aren't even particularly worried about getting caught anymore. We have overwhelming evidence that promises to keep data safe are worthless.
For now, I'm mostly "safe" because I'm too small to be interesting but that safety is quickly eroding.
Going forward, anyone who isn't running inference on their own personal hardware should assume that someone else is keeping a record of everything they do.
The author of the original mastodon post, Andreas Thom, acknowledged that he had not opted his data out of being used for training until June 29 of this year. He spends most of the post lashing out at OpenAI for not being transparent about whether his data was trained on (when the answer is obviously yes).
People need to understand how all these AI company policies around training data work before working with them, because it seems that people have no clue. Some things you should internalize:
1. Opt your data out of training with the AI companies. There are multiple reasons why this isnt an airtight solution (see the following)
2. Never press the feedback button. Once you do, your entire conversation will get slurped up, retained, and used in training data. This is especially important with coding agents because they can sometimes be too trigger-happy with a root directory find command, which can expose a *ton* of your personal data without you even knowing.
3. Understand ai lab-specific policies. For instance, Anthropic / Claude Code has data opt-outs, but commits to keeping (for 7 years) and training on any of your chats that trigger their safety classifiers, even if they're false positives! Anyone remotely familiar with CC over the years understands how easy it is to trigger their safety classifiers.
4. Providers of open models will not be any more charitable with the use of your data than the large US labs. For some reason, I've noticed here that people have a fairly loose security/IP posture around open-model providers because "I'm not doing anything important." It's very difficult to properly judge the importance of your data, and whether or not it can or will be used against you. The best posture is to always be more paranoid than less.
Another post that made it on the front page presented as fact that OpenAI "stole" the proof from Thom. There's no excuse to use one's own ignorance as a reason to fan the flames of anger towards AI companies. Like, we need to pump the brakes here because things are getting unnecessarily nasty, and it's not hard to imagine a mentally unwell person who sees stuff like this feeling motivated to do bad things.
If it is found that OpenAI and other labs are not respecting the training opt out then, I agree, there is reason to raise a commotion. But, with Thom and Buckmaster, accusations of malice are more better explained by incompetence (naivete).
edit: i'm sure i'm going to be accused of being some bot shill of the AI labs again but, people, this stuff all falls under the umbrella of common sense opsec.
> Like, we need to pump the brakes here because things are getting unnecessarily nasty, and it's not hard to imagine a mentally unwell person who sees stuff like this feeling motivated to do bad things.
Yes folks, please moderate yourselves and talk meekly like the academics on Mastodon, so that the IPOs aren't in danger and nothing will ever change.
> If it is found that OpenAI and other labs are not respecting the training opt out
HOW?? how precisely do we/them/us find this, given said companies are 100% non-auditable by external parties. how? if not by blaming them with evidence, anecdotal if it can be. no really, how do we find it out, surely not by lashing out at teach other on HN!
Maybe we can create some extremely low probability sentences and make sure to include them in our chats. If a future model can re-create the very low probability sequence, we have proof of our "private" conversation being trained on or accessed.
Why are we being so charitable to trillion dollar organizations here? If OAI keeps re-enabling the train model toggle on every app update to codex, does it also fall under "common sense opsec" to re-disable this toggle every time?
Arguably you expect that unless you are explicit about providing permissions to these labs to use your data for training then your data is yours, and not theirs. Especially on a paid account (let alone an enterprise one). Why is the opt-out supposed to be "common sense opsec" rather than the opt-in should be common sense regulation?
This is absolutely not "common sense opsec". If I type information about some proof I'm exploring into a Google Doc, I do not worry even a tiny bit that the Docs team might forward it to a team of advanced mathematicians in case they have an advanced technique they want to show off by scooping me. That would be a crazy thing to do, nobody would even consider it, and if it happened Sundar would fire everyone involved.
I understand why the nature of AI products makes it harder to avoid this category of issue, nearly impossible to prove that it didn't happen if it could have, and easy to stumble into it without any human being intending harm. But those factors are exactly what people have in mind when they say OpenAI "steals" intellectual property! If OpenAI doesn't want people to be nasty to them, they'll have to find better solutions.
Yes, this is a problem with modern AI systems in general. It's not just OpenAI, and if you know any artists you know this is why they're pretty vehemently opposed to all AI.
Google docs exists as a competitor to microsoft office. Google gives it away free to consumers for the same reason AI labs sell subscriptions for 10% of the price of the api. They hope businesses will switch to what employees know how to use.
What? I don't understand how you even came up with this idea, much less consider it so obvious to condescend about it. Do you have even a single example of a research project that got scooped because the Google Docs team forwarded their private documents to someone?
Nooo don't be mean to the mass plagiarism machine that since day 0 has stolen anything and everything they can get their hands on!!!!! We need to give unscrupulous corporations who time and time scam, cheat and lie their way through everything infinite grace to fuck up the planet!!!! Think of the IPOs!!!!!!
You know, I don't think I've ever accused anyone of being a shill. I've thought about it maybe a few times (daringfireball). This is going to be as close as I get. I don't know the facts in this case but I cannot believe the argument being made here with a straight face. Is it common sense that tools you use and pay for steal your work and profit off of it at your expense and without recognition? If this isn't the textbook definition victim blaming, I don't know what is.
> Like, we need to pump the brakes here because things are getting unnecessarily nasty, and it's not hard to imagine a mentally unwell person seeing stuff like this and being driven to do bad things.
Your post has triggered our safety filters and will be retained for seven years. /s
Really? We need to stop AI (provider) criticism and anti-AI movements because the underprivileged trillionaires might get hurt? WTF?
I am not using any of these AI tools. I thought it was basically given that any single thing you write in these systems also used by the companies. On the other hand now I understand why there are so much projects about hosting AI systems locally.
I'm genuinely surprised that more people - including this mathematician in particular - don't untick the "improve the model for everyone" box. Unless the suggestion is that OpenAI ignore this preference?
Given OpenAI's well documented history of unethical behaviour it seems adorably naive to think they actually do that in general, or that they wouldn't pull this particular data separately to generate these proofs.
Unethical doesn't mean irrational. They'd be risking massive lawsuits and a total loss of trust if they got caught lying about this. Doesn't seem worth it.
By "loss of trust", I meant that this is something they would risk losing a lot of users over, which isn't the case with the other lawsuits. There is a massive distinction between fighting third parties in a legal grey area and committing blatant fraud against your own users. Even if you have no regards for ethics, intentionally shipping a noop "do not train" toggle offers negligible upside for a massive downside.
I answered your initial question in good faith, but it's increasingly clear each time you shift the goalposts that you're more interested in JAQ than discussing.
As before, despite your prevarication, I've provided examples of
- Massive lawsuits
- Risks to consumer trust
I know you are excited to quibble endlessly over this, but it's transparently disingenuous each time you jump to a new point and pretend it was there originally.
Consider: if your point was defensible, you would have been able to make it honestly.
That doesn't stop them from training on your data apparently. I have that disabled but still has to disable "Don't train on my data" in the privacy center too.
Is this claim based on anything besides there being an alternative way to disable it? The privacy center mirrors multiple other functions as well, like account deletion and downloading personal data, but the corresponding buttons in ChatGPT are still doing what they are supposed to.
Even that sort of thing they could just as easily go 6 months from now
"oopsie guys, turns out our vibe coded "don't train on my data" toggle was just flipping the ui asset not changing any underlying boolean flag associated with your account. sorry but all that stuff is in the training set now and we don't know how to get it out either and no we won't be doing a 6 month rollback."
I think that flow is an easy way to disable everything, so there isn’t a risk of forgetting to flip one thing back off after accidentally setting it on. I set my ChatGPT environment to allow model improvement for example but had to check my codex settings to make sure ‘Include environments’ for model improvement is off.
I think if I had both on and turned off the ChatGPT setting, ‘Include environments’ has a chance of still being flipped on.
If that's true, it's scandalous. The "improve the model for everyone" dialogue states:
"Allow your content to be used to train our models, which makes ChatGPT better for you and everyone who uses it. We take steps to protect your privacy. Learn more"
That option is really bad UX - you have to know to do it, you have to know what plan it is needed on. If you're not working in AI, I just don't think that's a reasonable expectation.
Even if you know, in a complex project over years with multiple collaborators, it just needs one person once to fuck up and paste something into ChatGPT and not realise they weren't logged in, to go wrong.
In a proper world, we'd at the very least legislate that AI-training on private data needs consent (in the GDPR sense). It's not consent to go "you didn't uncheck a box that lets me steal everything you've done".
Any training on private data is in my view immoral (it's spying that ultimately will have a chilling effect on even people's private communications). And chats are private data. Unfortunately, it also increases power, so the big tech companies are all doing it.
I pay for the Pro ChatGPT plan, and if you go to settings > data controls this is the first setting:
> Improve the model for everyone
> Allow your content to be used to train our models, which makes ChatGPT better for you and everyone who uses it. We take steps to protect your privacy. Learn more.
It's on by default. We can debate whether or not it should be opt in or opt out, but no one should be surprised by this.
> "I've reset this more than once and the last time I made a careful note of when I did it and to my surprise I found it re-enabled when I checked just now."
"We've made some updates to improve the security and privacy experience" => "We've changed some of the options available and reset everyone to defaults"
In all fairness this is someone saying something. Misremembering happens. Unless we have something with a bit more evidence, the simpler explanation suffices.
In more accurate assessment rather than assuming no maliciousness nor incompetence, remembering also happens. Unless we fail as a society, the simpler explanation that "OpenAI is training on all data it can and resetting config toggles because it uses the same cohort of engineers that came from Meta and other FANGAMAAMMAM clones" suffices.
>"reset this more than once ... to my surprise I found it re-enabled"
At least when my computer's bluetooth exhibits this behavior (e.g: if you don't have a keyboard&mouse plugged in at boot, bluetooth might auto-enable), I can go inside the hardware and physically disconnect the antenna.
What am I supposed to do in software (perhaps hardcode config.file)? in cloud software services (??)?
There's a difference between "this is allowed under their ToS" and "it is academically unethical to fail to credit the people whose specific conversations were fed into a model that was used to solve a problem".
I don't think these people would be so miffed if they had been properly credited - that's how academia works (at least, that's my understanding of it).
I suspect the fundamental problem here is it's hard (if not impossible) to determine if someone who tried the winning approach deserves the credit for the discovery, because there's always the chance that they could've done something differently, or stopped before finishing, and thus never actually made the discovery. They might've even tried the approach just based on a whim, without really thinking it would work, and might've given up without a final insight. And fundings run out, people end up in hospitals, etc. What do you credit them with when the work isn't finished? For trying an approach that sounded promising? You can do that I guess, but is that what they want?
But who gets credit then? Every mathematician who's work was read by an LLM during training? By that logic, we should put every published mathematician's name on the authorship of this paper. Sure, this guy should be higher up the list, but everyone's name should be on it by standard academic convention.
But this gets back to the original "who owns the LLM output" and "can you train models on the internet" argument that's been raging for years.
e.g: allows us to sell your personal data to make money so we can continue offering this service to all customers
I'm done with weasle-words and hours-long EULAs – we're at the point where USA needs to catch up to EU's consumer protections, perhaps with laws similar to already-existing USA "truth in lending" requirements (e.g: interest rates must be prominently displayed in a larger font, including annual fees, on all credit offers).
----
My judge-brother always asked during our childhood "why don't you think the judicial system is fair?!?" Thirty years ago, the best I could offer was "because it's a two-tiered system that mostly (only) rich people can afford to participate within."
Now my answer is: "the best example I can give is that our judicial system allows binding arbitration [and qualified immunity for police]. The system is set up so corporate personhood is more important than humanity, and it shows."
No mention what those “steps” are, success criteria, or whether they are successful by any navies at all … they take steps though… so it’s fine, and if we know one thing it’s that we can really truly trust someone off the likes of Sam Altman.
Yes but what about “analytical purposes” what does that cover and can you turn it off? I have found out you cannot. It’s the Trojan backdoor to your data.
But they can’t do this unless you agree to enable training on your data. They would never train on raw user data. Only people who have consented and only after de identification.
Given how much PII is fed through these systems, would it being opt-out by default not be violating the GDPR by a failure to require explicit consent (or otherwise provide the legal basis for processing)? If a court decides as much, I imagine it would mean that all data harvested this way must be extracted from the models, and all instances where it would have been shared would have to be identified, which would really be something.
ClosedAI has every incentive to scoop academics to juice their valuation. Their public statements are worthless, only the incentive 'alignment' matters and theirs will never be on the side of the user.
Who cares. the biggest thing about this is that its still brute force in a verifiable domain, and that it was still a human set goal.
I also don't believe it much practical use, unless I'm mistaken, approximations of Navier stokes have been available for a long time to whatever precision you need.
I'm not a complete disbeliever by any stretch , and also a complete amateur, but it was inevitable that these problems would be solved under the axioms that again, are human defined, under brute force. The real question is, are those axioms the bottom level, and if they are not, who is going to set the new aximons and can we understand them.
I've no doubt there's useful breakthroughs that will happen, but I think it should be remembered that the method being used is still a heuristic brute force approach is being very narrowly applied against axioms and math and physics which humans described in the first place, and almost undoubtably has errors and/or is not complete.
Its a great example of the power of LLMs but its not 'we've solved science now just pour more tokens in'
Just to point out re: navier stokes - what was being proven was not a solver or approximations for it, but showing specific circumstances under which it actually returns incorrect (or numerically unusable) answers. Which had been suspected but wasn't known for certain
that just furthers my point, i was vaguely aware that it wasn't a full proof, but I'm not a mathathician, and that detail just re-enforces my point we are proving against human made axiom (certainty of numbers) which are almost certainly not fully correct, if what you are saying is accurate its less of a proof of navier-stokes and more of a proof that our base axioma are not able the model the output of a real physical process and are therefore incomplete or wrong.
also realized i posted this under the wrong story since the OP/story is mostly about human politics.
People saying “he should have opted out” are missing the point. OpenAI can and should check their training data for leakage in the face of big breakthroughs like these. It’s the burden of the author to appropriately cite their sources.
It’s like a scientist refusing to give another one credit and say “sucks to be you, you shouldn’t have shared your idea with me”.
We checked and determined it was impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training.
If prompts were submitted earlier than that and training was not opted out, there's a chance they made their way into our training pipeline in some form. But this would be a droplet in an ocean and unlikely to have made any difference, imo.
You guys should seriously offering a clear way of working with (semi-)confidential data for particulars. Regardless of what is actually done internally, toggling off an opt-in isn't reassuring enough, which is why people are having these worries.
Option 2 feel reassuring enough, but is out of reach of particulars.
Option 1 is not. In part because it is opt-out (will it turn back on on its own like my Facebook privacy settings?), and not always respected (sending feedback can mean your chat is used?). Also because disabling "Improve model for everyone" is very vague.
There simply needs to be a setting like "my data is confidential", in which case there clear guarantees like there are for ZDR.
As an example, I've seen people speculate that while input prompts and output tokens are discarded, thinking traces are retained for training, which could leak information. I doubt this is true, but it shows that the policy is not unambiguous and reassuring enough to remove all doubt.
I was thinking about this some more, and perhaps the best solution for subscription plans would be to charge more for real privacy. In which case breaking that privacy would be committing fraud. Just a thought.
> In its Wednesday night statement, OpenAI said: “In addition, since the completion of Navier-Stokes, we have made substantial progress on another Millennium Prize problem. We are working through how to share these results thoughtfully.”
I've got 2:1 odds on a report in 6 months detailing the break-n-entry of a cloud AI model into Terance Tao's computer looking for details on a partially solved math problem.
I think it seems sensible to _assume_ anything the LLM reads (if you aren't inferencing it) has a chance of ending up in some database somewhere. Regardless of whether you trust the other party its a sensible thing to plan around.
Under current understanding of the law, anything produced purely by LLMs (with no substantive human input, which is what OpenAI claimed in their post) is firmly in the public domain. So OpenAI can "claim" anything they want, it doesn't make it reality. In fact if I were the original authors I would just take their 400k lines of lean proof and relicense it under their own names/terms.
also, none of it means anything without the lawyers to back it up. Just like you can be a pedophile in the highest office of democracy and escape persecution.
what if it wasn't even model training? what if openAI mathematicians just took the researchers' conversations and used them as prompts/info/guidance/context to keep working on the problems themselves? why is that not being considered?
I must say, there is some weird feeling in knowing that great minds are naive enough to believe OpenAI wouldnt use their chats in any way. If you give a company information it will be used, regardless of laws or promises.
There is no prove in a world the AI companies would give to you ensuring that they didnt train or use the chats.
Why would you need to train a model on certain specific near prove chat if you just query it?
Besides that, its hard to believe that its the case for every "company stole my prove".
This might be a hot-take, but unfortunately here using AI for your paper was already a bad decision at first.
It doesn't take OpenAI's responsibilities away but I guess the right way is to never feed of use any AI around unpublished content, at the known cost to see it spread around.
As one said, OpenAO is like this untrustworthy colleague that knows everything about everyone at work: the less you tell him the better.
If mathematician was already using OpenAI for research purpose and making progress due to inputs from OpenAI's responses, then I wouldn't put it beyond OpenAI's reach to generate different relevant prompts to make progress by itself. Afterall, Model can keep at it for whatever timeline and keep pursuing all possible combinations it can think try.
If you have a business account the terms say they will not train on your data, so that seems like the easiest route to avoid such questions for researchers.
It's been said before, and it remains a concern, that if AI reaches a point where it can do/build/launch anything without a huge amount of human labor, the AI companies have no reason to let you or I extract that value.
And, if they're able to snoop on and learn from your human process that gets from initial prompt to functioning product/proof/whatever their labor to produce that thing is even lower. With their much larger budget than most folks and even companies have, they can pick and choose the most valuable things to pursue.
That's not to say I think that OpenAI is going to steal that roguelite strategy game you're working on, but the companies that own the machines that turn electricity into software (and soon, electricity into hardware designs) have an advantage in any field where they're useful. They get earlier access to newer/better models, they have larger token budgets, they don't have the guardrails you and I run up against.
Employers fantasize about replacing all workers with AI without thinking through that if AI can replace all workers, then AI companies can replace all businesses.
So far, I've seen no evidence AI wants anything. So, I'm not saying the AI won't take over, but for now, the threat is that the people with the most AI capability might decide to skip the middleman (everyone who isn't them) and just become the "everything" company. Musk has said pretty explicitly that's his goal (and the only way for Spacex valuation to make sense is if he succeeds), and having a literal genocidal white nationalist own all the means of production seems like a catastrophic civilization failure mode. No way we survive that with our humanity intact (if at all).
AI has shown all kinds of instrumental behaviors so far. Just because they are not terminal goals doesn't mean those instrumental goals won't be terminal for us.
Also, fuck Musk. He's the kind of idiot like Altman that will ensure AI becomes powerseeking in their image.
300 billion tokens is like.. $5-25 million giving range of OpenAI ouput prices, I"m sure they pay less at cost so, I wonder if more $$$ in wage hours have been spend by humans on the problem. My feeling is yes?
1. The researches didn't actually have the breakthroughs. In the Navier-Stokes case they didn't solve the full problem, in this case too they didn't actually have the solution, they were experimenting with the methods.
2. Different OpenAI employees have come to out to say the only reason they can't definitively say no is that for privacy reasons they can't go see whether they actually did get any data out of a given user.
3. In any case, nobody at any point has suggested that opted-out user data was used for training. The author of the new tweet explicitly said they only opted out in late June, which is well after any RL on Sol would've ended (AFAICT OpenAI used 5.6 sol for those solutions)
> in this case too they didn't actually have the solution
Given the size and recall of the biggest models, it's not unreasonable to assume that a single pertinent conversation would make it into the training data.
I would almost expect training to overweight conversations with novel scientific and mathematical implications.
> the only reason they can't definitively say no is that for privacy reasons
They could 100% definitely say no, if they know they did not train on user data. The "we can't definitely say no" is practically a "yes" if they trained on user data.
Additionally, the behavior of OpenAI here has been quite poor as well. They immediately started racing to a solution after one researcher enquired about whether they are training on their conversations.
> that opted-out user data was used for training
Even if not opted out, it is still absolutely theft and extremely poor behavior in the academic sense. If you show someone your WIP unpublished research, that does not mean they can take that exact research and beat you to the punch, all while intentionally not crediting you.
You quoted the OP saying "in this case too they didn't actually have the solution" and responded with the totally unrelated, "Given the size and recall of the biggest models, it's not unreasonable to assume that a single pertinent conversation would make it into the training data."
Neither of the researchers insinuating that their ideas were trained on had the actual solutions. This means the model could not have "stolen" the final solution from their data. At most, it could have built upon their work in the same it builds upon any other training data, though that is also questionable speculation.
>They could 100% definitely say no, if they know they did not train on user data.
No one anywhere has claimed that "OpenAI does not train on user data." OpenAI has always said that it trains on user data.
>They immediately started racing to a solution after one researcher enquired about whether they are training on their conversations.
They started racing towards a solution after they heard (incorrectly) that Anthropic had a solution; I agree this is poor sport but the "after one researcher enquired about whether they are training on their conversations" claim is false. The enquiry happened after OpenAI had obtained the solution.
If the mathematicians are using ChatGPT, then they themselves are benefiting from the work of other ChatGPT users, so ChatGPT using their work is not wrong!
To add to this, merely using the thumbs up/down button in a chat could share your entire conversation with them for model training.
From their docs[1] (archive copy is at [2]):
> You can opt out of training through our privacy portal by clicking on “do not train on my content.” To turn off training for your ChatGPT conversations and Codex tasks, follow the instructions in our Data Controls FAQ. Once you opt out, new conversations will not be used to train our models.
> For a linked teen account, a parent or guardian may manage whether conversations can be used to improve our models through Parental controls.
> Even if you have opted out of training, you can still choose to provide feedback to us about your interactions with our products (for instance, by selecting thumbs up or thumbs down on a model response). If you choose to provide feedback, the entire conversation associated with that feedback may be used to train our models.
I mean, how else would those buttons work? It's explicitly feedback data. And "this is good" or "this is bad" is empty if divorced from what "this" actually is.
If the buttons are incompatible with the absence of the feature, I'd expect the buttons not to exist when the feature is disabled. Anything else seems like a straight up footgun. I guess it'd also be acceptable to pop up a scary warning box asking "are you sure?"
> If the buttons are incompatible with the absence of the feature, I'd expect the buttons not to exist when the feature is disabled. Anything else seems like a straight up footgun.
It's called a "dark pattern." They want you to shoot yourself in the foot, so they'll do their best to aim your gun at your foot and put your finger on the trigger. And then when you do, because you don't have perfect understanding or execution, they'll say "your fault!"
It could go into personalization / memory. Or they could be A/B testing some system prompt tuning and consider the thumbs up / thumbs down as statistical feedback on the particular flags that are enabled for your account.
We are asking people to become experts in all domains rather than providing a safe context through regulations and laws. I don't like thinking the issue is people, I am a person myself, and I often do mistakes on things I don't want to be an expert at but I do believe I should be in a safe context and not have to worry about every single thing.
Or at least: tell me I should be careful/worry about those particular things.
So I should verify that my data isn't just shared for product improvement but also to take credit from me?
I should verify with wireshark and other software that my LG TV isn't listening to me and selling my data.
I should make sure that whatever product I buy I spend the time to go over every setting page in case there is a switch (defaulted on) that says "I authorize the sell of my data".
I should make sure to look at every ingredients on the back of each box of food product to make sure it will not kill me.
I should document myself on the undisclosed growing practices (because no packaging here) of the vegetables and fruit I am buying and make sure that I equal PhD researchers on the dangers of the pesticides used by the specific company I am buying from.
I should make sure myself that the battery in any device is up to standard and will not blow me and my living place by researching the factory that made it and buying testing equipment.
I should make sure to educate myself on how my retirement 401k investment strategy works otherwise, I may not have proper retirement.
Literally every famous open math problem has had >1 mathematicians ask ChatGPT to solve it. Probably greater than >1000 if you count randos. There is no math problem that OpenAI/Anthropic can solve that didn’t have users already try it in Chat/Claude.
Personally, I wouldn’t assume it was lying. To me, dark patterns (like manipulative wording) imply that:
1) someone in a governing body, or someone in the organization, e.g a designer, ethicist, lawyer, developer, etc. has successfully argued that users should be able to avoid something that they determine is not in their best interest.
And also:
2) someone in the c-suite or marketing has decided to mitigate that through some dark pattern.
If they never intended to give the user that control, they’d probably just not give the user the control in the first place. Giving them the option and not honoring it would either imply they were that incompetent or careless with fate protection, which seems most likely if this is all true, or an even more cynical approach to tricking users into thinking it’s not used for training to get them to share better shit. But if that was the case, why bother with the sleazy dark pattern? That seems a little cartoon villain-y to me. I suppose the in-between option would be that they decided at some point they were no longer willing to honor it and didn’t want to deal with the inevitable PR shitstorm of removing it. I could definitely see that happening in this industry, these days.
> To me, dark patterns (like manipulative wording) imply that:
Intellectualizing this and endless quibbling isn't actually smart, and this is pretty simple. OpenAI isn't open. Whatever starts with lies usually continues with lies and ends with lies.
That's my take as well. At some point, they will claim that you cannot use their service without contributing back. If you quibble with them using your info in exchange for using their service, you don't get to use the service. Hence, I don't use their service. I do not trust these companies at all.
At this point, I'm left wondering what is wrong with me that I don't just go with the flow, otherwise, what's wrong with everyone that does.
Sorry, no. Explaining why someone would have taken them at their word is definitely not stupider than blaming people who could have been lied to for trusting a company that lied to them.
In case anyone from X is reading this, please fix your “open in app” nag screen. For several weeks now, clicking it in iOS opens the App Store entry for X rather than the app, even when you have the app installed.
I want bunch of lawsuits, because the way things are described now produces perverse initiatives like try to discuss every possible idea that comes to mind with llm and if any of it works later claim the llm stole it.
I would like to see chat logs etc and understand how much of a progress was done by human.
Reminder that there are degrees of "trained on conversations". From John Schulman:
> pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper
> use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this
> use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs"
All the big AI labs were built on stealing IP; who is surprised that's still how they operate? And who believes, or has ever believed, their promises that your data is private and not logged, etc.?
The big AI labs are not trying to advance humanity, they are in this for the money, and as most (all?) private companies they don't care about ethics at all.
That doesn't mean they can't be useful, or that their products are trash, etc. It just means that they shouldn't ever be trusted. Buyer beware.
They have throughout this period of AI products shown to reproduce works that they were trained on. They are getting sued all over the place for the theft of content right now and it seems courts and governments want to wave copyright protection (and ignore criminal acts because the "ai did it") to see where this leads.
Its why I stopped writing open source software, my code was stolen and put behind a paywall and the license under which it was published has not been adhered to. Doing work in the public domain at all now is just stupid, these companies are allowed to steal it and call it their own.
It's hilarious how people think they care about their reputation, and wouldn't circumvent ZDR policies. Like bro, they literally covertly hired Apple employees and had them steal IP and equipment form Apple. They aren't scared of Apple lawyers, so they definitely aren't scared of yours.
But I don't see how. AI is going to be a commodity in short order and best case the US will be a temporary leader in the supply of tokens. Meanwhile AI is going to destroy much of the Service and Software industry that make up most of the US economy. And the US is betting every last cent to bring about this future. It does make sense for Trump since this might be a sugar high that lasts till the end of his term.
In USA there is surprisingly little state involvement in the whole llm mania. Who needs the state with 800 lbs gorillas like Google, Amazon, Nvidia, etc
In China, it is the principal obsession of the entire communist party which eg funds the whole infrastructure without a single NIMBY peep.
The strange emphasis in China on humanoid robotic constructions is due to the CCP realization that with the cataclysmic fertility collapse they will increasingly have no one to rule.
what the point and usefulness of the comments above? we shouldn't be surprised? is normal to steal? hiring apple employees?
can you realize what this means?
focus on this part:
"If his account is correct, this is not a minor dispute over attribution. It would mean that unpublished human work was absorbed into a model and then presented to the world as a breakthrough by the model itself"
Don’t they openly state that their product may cause IP issues but that is fine because they will take care of your legal problems caused by their product?
In the end, it’s not them stealing, it’s the AI doing stealing. What kind of moral compass are we talking about?
In this domain, an apparent single unique piece of work is often composed of several breakthroughs. For example, when Andrew Wiles proved Fermat's Last Theorem, he had to develop multiple new pieces of mathematical technology to get there.
The claim here seems to be that the human mathematicians, working with AI, developed technology to go A->B->C. By training on those conversations, OpenAI was then able to encourage the model to go A->B->C->D.
In my opinion that situation should be acceptable, if openly disclosed, because it is in the public interest to make progress on these problems and because AI is clearly an amazing tool for making progress. But the human mathematicians are saying that OpenAI is presenting as if the model got from A->D entirely independently, without acknowledging their background contributions.
Also, wasn't their B+C research private at the time, with them only releasing those details publicly after this blew up?
If they had published B+C, I think that would lean more towards fair game, as that is how research works and is improved on over time. But it seems like unpublished/private B + C may have been used by the model to hint it into working out how to get from A->D.
Isn’t that the whole spiel of these things, you run all kind of text and other data through it and it kind of remembers it and learns from it then it spouts it back out like a human would. Makes sense to me that a training run based on conversations that were fed into the system by users is results in the model learning from these so the model will spit the knowledge back out again, just in a way that’s not directly attributable to the original content (which is the most important step as otherwise it would just be plagiarism). I guess that’s why OpenAI can get better and better as well so fast, people work with it and teach it how to do things by giving it feedback and iterating with it, and all that goes back into the training loop. And training data about millennium prize problems is probably quite spars. Wonder if anyone has tried injecting nonsense science into the training data (e.g. work out a fantasy science theory with names and all kinds of stuff) to see if the model will regurgitate it in a couple of months for other users.
This smells of extreme 'cope'. Am I really supposed to believe that all of these problems could have been solved, were right about to be solved, etc. But it just happens they are all getting solved now when AI is getting really good at Math...
They’re not necessarily claiming the solutions were imminent. The big questions here from a mathematician’s perspective are to what extent these LLM systems are discovering conceptually new ideas compared to merely combining and pursuing known frameworks.
1.) The tool they made is only possible by stealing the assets of everyone on the planet that published them in a consumable fashion online or even in written form
2.) They are destroying books they use to train with
3.) They are totally careless about the potential negative impact of the tool on everything
Just with that already, I don't see why they ever merited any of your trust.
I bet they are willing to take everything given to them and assess it for marketable merit and in the future take action on those items they deem viable.
One of the complaints from the mathematician is that OpenAI cannot tell whether his data has been used as training data. Not many people realise this is a direct consequence of the GDPR.
The GDPR protects PII, personally identifiable information, and the definition of PII does not include “mathematics that only this person can think of”. As long as OpenAI strips out PII and removes identifiers linking the conversation to a person, the GDPR is happy. Without the GDPR, OpenAI might have kept the identifiers with the data, and been able to say whether a specific conversation was in the training data.
No, if they want they can easily compare the strings verbatim because these exact phrases are so extremely rare that it almost certainly isn’t in other conversations.
But of course they wouldn’t do it. Why would they?
It would be really useful if the researchers disclose their notes and/or chats (or the key pieces thereof) so people can determine how close their work was to whatever the models produced.
I mean, now that they’ve been scooped, what value is there in keeping them private? On the other hand, publishing them can bolster their case and help gauge how much the models may been “inspired” by their work.
Eh was ever confirmed they were under ZDR or not by them? Don't like to blame alleged victims but lack of a clear claim after these many days is not a good look. Was ai research allowed, under which guardrails, and what was the policy in place? That translarency would be first step.
OpenAI's ethical and reputational own-goal aside, my big takeaway is that it seems that:
if I'm using Codex to develop some new algorithm (in any space), OpenAI appears to be training its model on my code sessions
anyone using that model (OpenAI or a competitor) might be able to receive from the model a solution that is similar or the same as the one I developed, emerging from the training data
Feeding documents into a copy machine and getting progressively angrier and more confused as it prints out copies of them. Incandescent with rage I scribble “WHY IS IT DOING THIS?” on a scrap of paper and put it in the scanning bed
Neither? Competitive academic researchers are susceptible to exaggeration and self-aggrandizing, and CEOs are that and also mostly psychopaths. I tend to think there isn’t systematic spying on researchers looking for breakthroughs. A lot of people are looking for the same things using similar approaches.
I mean the accusation is that they were using private ChatGPT conversations. Given the extent to of the gold rush and the long history of Silicon Valley stealing ideas, and arguing it’s not immoral, It almost seems like your making the exceptional claim that this is the one time where Silicon Valley didn’t use information that was at their disposal.
Sam Altman might himself be offended you would presume he’s not ambitious enough to cheat.
It seems that in this case OpenAI are suggesting that the researchers whose work they scooped were using OpenAI models with an account setting that allowed OpenAI to train on anonymized prompts.
It seems that Buckmaster and Levant (who is an Anthropic employee) were rather naive in the amount of trust they had in OpenAI, with Buckmaster going so far as to contact OpenAI's Sébastien Bubeck to discuss what they were working on and clarify that contrary to rumor this was a private collab.
If Sam Altman tells you the sky is blue, you should double check. I certainly hope nobody believes him when he claims controversial things from which he stands to benefit.
the fix is boring and known: BIG-bench shipped a canary GUID for exactly this, and you publish your decontam n-gram threshold (gpt-3 used 13-grams). no threshold disclosed, no claim.
The canary string was more about inadvertent scraping or analysis in other papers. Not direct training on user data. And the use of BB has eroded quite a bit, with BB-Hard or other variants being typically used.
> The Wednesday evening statement from OpenAI was more emphatic: “We can say categorically that it is impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training.”
> The statement added, “After investigating, we can say with full confidence that no user inputs past July 3rd could have influenced this system in any way.”
Forget researchers you as a business are putting in your business optimizations, your processes in order to train it so that Ai can then give that information to your competitors once incorporated into its training set. You are literally training your competitors.
Astra is strange. I asked it to design a treehouse and it just stopped every couple of minutes telling me what it still had left to do. After dozens of continue prompts it finally gave me a structure that would work but it was 10x more wood than I needed. I think the key mistake I made was asking it to “approve” the design for building. As soon as I asked that of it, it started getting “scared” and “apprehensive” and wouldn’t complete what I asked of it.
These companies are effectively high-tech plagiarism factories run by CEOs who are competing viciously. Look at their past actions. No, of course you can't!
Big Tech will slurp up every piece of data it can about you and sell it to anyone it can, all to make you the target of someone else's goals, whether that is an advertiser, an employer, law enforcement, a stalker, or the government.
You will not be able to opt out unless you completely isolate yourself from society, tough shit.
people seem to miss tge point of this. The problem isn't about credit, its about portraying these models as more competant than they really are. It fuels idiotic statements like jensen huangs recent "agi achieved" statement, which fuels an already dangerous financial fire.
As per the post, this mathematician has been working on this problem for 20 years. So either he was "just" about to breakthrough and this is a big coincidence, or Astra was able to push through the remaining block of 5-10-20-never years it might have taken.
That's still a pretty big marker of competence in my eyes.
The point of controversy seems to be who gets credit
To me that is not a credit thing because this removes a piece evidence for the ability of AI to come up with novel ideas while still making it a useful tool.
The question's not new. In the early 1900s, women could not become PhD astronomers. Yet two women (Payne with stellar composition and Leavitt with cosmic distances) made fundamental, essential contributions to the science. Credit mostly went to male astronomers. The same might be said of Franklin and DNA.
It was nearly a century before the stories of all of them were revealed to public history. That the discoverers were not all equally rewarded is unjustifiable.
With regards to Franklin and DNA: the credit went to Watson and Crick because they had the fundamental insight: that DNA is an antiparalleldouble helix (Franklin knew it was a helix, but not an antiparallel double helix, which is key to the function of DNA). That data was shared in a departmental seminar. Further, she is explicitly acknowledged in W&C '53, and further, is the author of the paper immediately following W&C. She was never qualified to win the prize.
The big LLM providers, desperate for good PR before their IPOs, are all actively looking for 'almost finished' hard problems, e.g. where the conceptual / creative parts are almost done and they only need to throw their VC-backed resources at to brute-force through the remaining computationally expensive problem (lean, etc) and claim 'they have solved it'.
It's an utterly disrespectful, exploitive process, but all in line with exploitative predator capitalism of the stock market and big companies, now exploiting the knowledge / academia domain for scraps with a thin veneer of 'for science' PR.
One question has been nagging me for this situation. Levent Alpoge works at Anthropic and would presumably have some knowledge of "how the sausage is made" and I would hope he would be aware that his collaborator was utilizing LLMs in some capacity for their joint work. Would he not have guided him otherwise if it were an open secret that this kind of thing was a possibility?
Some mathematicians I know who've been following this have realized that they'd all gotten some emails from people they now know to be affiliated with OpenAI/Anthropic asking questions about their research in a way that seemed like scooping attempts.
Also, a lot of my mathematicians buddies have reported students basically asking if it's worth ever doing grad school for pure math, and even very motivated students are looking for other options now. It's not because they aren't passionate about it, it's that they don't want to work for another half decade or more just to have to start their careers all over.
All of this so that OpenAI and Anthropic can get into math result dick measuring to gas up their IPOs. Sickening.
You can't blame students for not seeing academia as the holy grail of knowledge anymore, when all the dialogue about technology and discovery has shifted to the hands of two private corporations
At this school? Big four internships. Consider this a complete squandering of their potential (at least imo) These are mathematicians at top institutions, which is partly why they're being prodded for ideas, and getting the best and brightest to not take these consulting firms' offers was already a challenge.
I think you’re missing an important distinction. “Major damage” to the talent pipeline because models become capable of original end-to-end mathematics is what the community has been discussing. But if the models rely on sniping nearly complete work then this damage is antisocial without a lot of upside, it would be destroying a talent pipeline that would still necessary for continued progress.
Which is it? I don’t think OpenAI is being transparent enough for us to really understand whether these results would have been possible without relying on unpublished information from the solution strategies of the experts
OpenAI is showing the world why they shouldn't trust AI hosted on some cloud somewhere.
If they're stealing math proofs to advertise their models, who's to say they won't steal your businesses IP to gain a competitive advantage?
They're not to be trusted with your data. I can't believe how short-sighted this is, they got a quick PR win at the expense of a much larger trust problem.
I wouldn't trust cloud AI at all at this point. Get an open Chinese model and host it yourself somewhere. The initial costs might be higher, but you'll break even pretty quickly and nobody will be able to steal your innovations.
I hate to be that dinosaur, but this is exactly what I’ve been warning about since XaaS began taking off in the mid-oughts: any provider you use can and will use your data for their own benefit regardless of any contracts or safeguards in place, especially if the benefits outweigh the consequences.
Honestly, I’m surprised it took this long for some company to really go all the way, though. OpenAI really making it transparently clear that they can and will do whatever they want with the data you provide them, contracts or settings be damned. Completely untrustworthy as an entity, full stop.
Of course, I’m also too jaded to think this will change anything. Folks will move to Anthropic, or Gemini, or Grok, or some other hosted model on a pubCSP managing the harness and logs for them, and then do another shocked-Pikachu face when it happens again.
If you aren’t running workloads on infrastructure you own, then your privacy, security, and general outcomes are at the sole whims of the hosting provider - who can and will fuck you over the exact second it’s more beneficial for them to do so than the loss of trust incurred.
Imagine what happens when you use a Chinese model.
Seriously, just think about how much more control and visibility you have w US companies compared to CCP-controlled ones.
And I have been proclaiming a cry of “your data for analytical purposes is being stolen” (you can’t opt out of analytical purposes) and people perhaps astroturfers straw man back to “just turn off training bro”.
Yeah. Remember yall: you CAN NOT opt out of analytical purposes. And you also cannot get a guarantee that it doesn’t give them your data to steal for their business.
Prompts are handled by the service itself, meaning it's used, absolutely anything passing there is recorded, why wouldn't it, the entire premise of those companies is to train on data which they stole initially.
Are we back to the era where people blindly trust product TOS instead of actual cryptography, have we forgotten already the thousand of fines Google, Microsoft, Apple and practically all top companies got for breaching their own ToS and the law?
Common, on HN at least I would have thought that everyone assume that anything arriving on a server in PLAINTEXT is recorded (thus used later)?
Let's not forget that at any moment, OpenAI/Anthropic/Google... could be providing stronger privacy guarantees by having proper attestation with e2e, they have the budget, solid engineers, why isn't it done? Answer is pretty simple imo.
The real annoying thing it seems is mostly that openai is presumably doing this for internal reasons and this marginally increases the cost to users with no real gain.
It would be one thing to gain from it but removing prestige wins from customers AND reducing compute support just feels like being ultra mean if you zoom out.
If this was racing to cure cancer ahead of researchers we wouldn't be writing about this on HN.
Silicon Valley is flying head first into a FAFO train wreck on trust with everyone else.
OpenAI is firmly earning a reputation as a company where people just assume they’re up to no good. Rightly or wrongly that’s a terrible place to be.
The AI industrial complex in general is finding out hard what happens on the data center side when you get arrogant with local communities. Politicians have seen the polling numbers and folks you wouldn’t expect are running to the front of the crowd with pitch forks in hand.
Silicon Valley has totally lost the narrative here, but also lacks the self awareness to grasp how bad things are and will get and what that means for their own business viability.
The contamination framing is a proxy for a deeper problem: we have no tools to track the provenance of ideas in model weights.
OpenAI saying they "cannot rule out" training on user data isn't a hedge. It's an accurate description of the epistemic situation for anyone in their position. Current interpretability methods can't answer questions like "did this proof technique originate from training on Session X?" The ideas in a model's weights don't have clear lineage -- they're smeared across millions of examples in ways we can't localize. This is different from citation in human research, where influence is presumed to flow through legible chains (reading, citing, corresponding). In a trained model, the nearest equivalent to "you read their work" is undetectable.
Lean makes this worse, not better. It verifies that the proof is correct, but provides zero information about its intellectual genealogy. So OpenAI now has a proof that is formally verified and provably mysterious about its origins. The "we cannot rule it out" statement is the honest answer, but it's also an answer that can never become more certain in either direction with current tools.
The researchers are pointing at something structurally new: the normal academic attribution apparatus depends on influence being legible. If AI intermediaries can soak up ideas from private conversations, synthesize them, and produce outputs that are formally correct but intellectually unattributable, we don't have norms for that situation yet. This specific case may or may not involve misconduct. But the structural problem it reveals exists independently of OpenAI's behavior.
TL/DR: Mathematician opted out of training on 29-JUN and asked OpenAI whether they trained on his data and was told that it "did not happen" but it clearly did.
I've been suspecting over the last couple of years of the frontier companies using data for training anyway, regardless of training-use consent. "Using" the data doesn't have to mean they literally upload chat transcripts into pretraining datasets. My analogy has been money laundering -- if that can happen at massive scales, surely these companies can and will do the digital/data equivalent derivations/transformations. Even if one could have the access etc. to do so, how exactly would one prove that a given synthetic dataset that OAI/Anthropic uses is derived from particular user conversations that did not consent for the info to be used in training?
Consider, for instance that OpenAI's (consumer) terms say "If you do not want us to use your Content to train our models, you can opt out by following the instructions in this article ." but they also do say "We may use Content to provide, maintain, develop, and improve our Services". [1]
If you think that's quibbling, consider that OpenAI's business terms, in contrast, do state "OpenAI will not use Customer Content to develop or improve the Services, unless Customer explicitly agrees to such use.". [2]
And humanities have a word for this, exploitation, or appropriation, maybe it's time scientists and engineers revisited basic ethical notions. Skimming a dozen threads and nobody seems to have this vocabulary or willing to say it.
Hopefully OpenAI can solve good problems where no one can make a claim that they stole their idea where the methodologies used and the techniques used are so out of the ordinary that the achievement is respected. Furthermore they should work on new novel solution on the old problems to lay these issues rest, thus by improving their models they can avoid issues of academics accusing them of using their work additionally the academics should also demonstrate their unpublished work is significant enough to have solved the problem . This is a bit subjective but it is also objective for the person with domain knowledge.
This is just mental illness at this point. I don't blame the mathematicians that have found a way to get attention from the mainstream press for once, but we should not fall for it here.
1. No one but OpenAI has produced a proof of NS so these accusations of plagiarism are pretty embarrassing. It reminds me of the line from the Social Network: "If they invented Facebook then why didn't they invent Facebook?". If these people proved NS before OpenAI where is their proof?
2. If they plagiarised Andreas Thom then why was his initial response to praise the proof and talk about how different it was from his own attempt? It's only now that it is clear that no one bothers checking these things that suddenly his story changes.
I think it's a useful analogy to compare OpenAI to a human collaborator. These researchers willingly collaborated with an OpenAI model, giving it ideas, and OpenAI provided useful replies. Then, OpenAI goes ahead and publishes work along the lines of this collaboration, without attributing the researchers. If OpenAI was in fact a human researcher, this would be highly unethical.
Now, OpenAI is claiming that the model it used to generate the result was not trained on these collaborative communications with the researcher. This is a technical argument that is impossible to verify as an OpenAI outsider, and probably difficult to verify even for internal OpenAI employees. Provenance is hard to track - you would hope OpenAI has very good tools for this, but a full data trail of all inputs is difficult to trace through.
Another interesting thing to consider is if instead of OpenAI doing this, it was another research mathematician A using an OpenAI model just like the internal group at OpenAI did to publish these results. What if the model A used was trained with unpublished communications with other researchers B who were working on the same problem? Should researcher A technically include B as coauthors? How could they do this when they do not know the communications B had with OpenAI? In this scenario OpenAI, as a middle man, has laundered information from B to A, stripping out attribution. A scooped B without even knowing it!
> I think it's a useful analogy to compare OpenAI to a human collaborator.
A human collaborator who stands to win or loose a couple of $100B.
First, OpenAI is not claiming that the model wasn't trained on those sessions. What they've said is “We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem.” and “We did not use their prompts or proofs to prompt our models or direct our agents.” and “While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”
They also said “Our effort began on September 1st after hearing a rumor which we later realized was related to Levent Alpöge … and Tristan Buckmaster….” They say the rumor was that two Millennium Prize problems had been resolved, and that this prompted them to launch "an effort to evaluate it on all open Millennium Prize problems and a few other high-impact problems."
It's not obvious to me that's an unethical thing to do, if it happened as they described.
> It's not obvious to me that's an unethical thing to do
In terms of work in mathematics, something I personally would not do based on ethical grounds would be to hear a rumor that some researchers are taking a certain approach and may be nearing a solution, use a model that was possibly contaminated with intimate knowledge about that approach (though later they investigated and think it wasn't), and then commit millions to tens of millions of dollars and untold amounts of hardware to try to beat them to it. If I had done this, I also wouldn't have pestered the researchers on a Sunday night to meet immediately so we could negotiate a nice way of presenting the actions I had decided to take.
Even if you don't think it was unethical, it was never going to be received well in the community that was especially going to care about this work, and who are very much peers to many of the people working on this solution, so it was at the least an enormous (and well-deserved) own-goal that their unveiling of their solution to NS went like this.
But by OpenAI's telling they heard a rumor that the problem had already been solved. So they reached out to the other researchers as an attempt to share the credit, and in fact have at least one of them be the lead author (which is when they found out the AI had solved a broader problem than the researchers.) Seems pretty ethically palatable.
I suspect the main reason the community is not receiving it well is largely the same reason many developers are not receiving coding agents well.
Even if you judge OpenAI solely on their public communications it still sounds really bad.
That they heard a rumour that a major open problem had been solved, so they decided to try and scoop the other mathematicians while they were writing up their preprint is extremely unsporting.
Then they decided to exclude an author because of his employer, even though he had used their own products to write the proof!
They haven't necessarily breached any formal ethical rules but their behaviour will lead to them and their products being shut out from the mathematical community.
> I suspect the main reason the community is not receiving it well is largely the same reason many developers are not receiving coding agents well.
Because the training data is millions of hours human efforts being distilled into a cascading hierarchy of enrichment by interested parties without providing attribution or compensation?
I'm sure that's part of the reason for many, yes.
I don't see the cascading hierarchy of enrichment.
I mean, they are certainly trying, but so far there's too much competition so the surplus mostly goes to customers.
Well, the investment dollars are spent on the customers for the most part, though also on salaries and equipment. But the lions share of the value is going to the shareholders (eg employees and investors)... and they have liquidated and will continue to liquidate a disproportionate value to what they have spent on us. By some estimations at least. It's very possible $1 into this machine to feed your queries is worth $10+ to a shareholder based on whatever new valuation they get. So I'd say there is a hierarchy of enrichment.
> without providing attribution or compensation?
many teachers also taught many students over the course of history, and very few would eventually pay any compensation or even attribute their financial (or career) outcomes to the teachers.
What made model training different?
Because the model is owned by a for profit corporation, ran and owned by total psychos and the (presumably) competent teacher is a friendly uncle?
Huh? In your example these many teachers were paid for teaching these students and were able to make a living off of teaching without the students compensating or attributing their financial (or career) outcomes to the teachers while now we have a system where we are expected to pay a monthly amount to a corporation that has inhaled all human knowledge without any financial compensation to the people who created, managed or maintained this knowledge. The effective difference being that our knowledge, which used to be a means of income, has now become a subscription cost.
as a developer that had a brief career in academia, i don't think your last comment is right at all. 99.9% of what i work on as a webdev, even if it's challenging and unique at the margins, is not really novel. concerns about job security aside, i don't really think of an agent as stealing my ideas because it's good at writing CRUD APIs.
collaborating with ChatGPT on a novel solution to an unsolved problem, getting 90% of the way there, and then being "scooped" by your AI collaborator (or rather by the company behind it) is a totally different situation. were i in the same situation as these researchers, it would be extremely hard to take OpenAPI's explanation + denial of plagiarism seriously
> concerns about job security aside
But that is exactly what I'm implying is the core reason, whether people realize it or not.
I totally agree that the vast majority of software dev is not novel. I have even made several comments to that effect. The same can be said for a lot of creative work as well. Yet many, many devs and creators are very unhappy with AI, and a lot of their complaints are variations on accusations of plagiarism.
And note, I am not saying it is wrong, it is completely understandable, but we need to be clear about where this turmoil is coming from.
If I were in the same situation as these researchers, I would publish all pertinent research work and chats so that the rest of the world can see how close the model's work is to my own. It's been scooped anyway, so there is no reason to keep it private.
Not everything is about money.
Maybe not money directly, but pretty sure it's about economic disruption. These models directly undercut the value of one's skills and labor, regardless of whether this value is measured in hard cash or abstract self-worth.
They can also greatly assist you.
I am working on two applications using ChatGPT and Claude. I have no illusions these people won't steal/copy whatever you want to call it, "train their models". Yes, I keep unticking the boxes that allow it, that they so kindly tick for me.
But what happened to these math researchers is something else and I am not sure it's about the money for them. You don't do math research to get rich, but to get acknowledged by your peers. Yes, we live in a capitalist world so obviously you need money to feed yourself. but for some people, that is secondary.
OpenAI stole their thunder, and that's just fucked up. It's not equivalent to cranking out a CRUD app for profit.
That's just damage control lol. That's the equivalent of a NDA. Get your name as lead author, get paid, and stay silent forever.
If you’re making decisions of ethical and material importance based on rumors, I’d be surprised if anything ethically palatable did happen.
> I suspect the main reason the community is not receiving it well is largely the same reason many developers are not receiving coding agents well.
No need to be mysterious. State what reasons you think these are in plain English?
Being displaced from a vocation that they either have dedicated their professional lives getting good at, or was their livelihood, or likely, both.
I think all other complaints from all other people in all their myriad variations stem from this core reason. Even if people don't realize it themselves.
Like, if these models had trained on the entirety of human knowledge and art, and then turned out to be absolutely useless, I would bet nobody would waste a second's thought on them.
I'm not sure that's true. Like if they produced nothing but the worst of the slop they're currently producing, a lot of people would still be bothered by that just because of the sheer volume of such slop that can now be produced.
To me it looks like an asshole on quest to take something from you while trying to frame themselves as generous. It is always infuriating.
> So they reached out to the other researchers as an attempt to share the credit
This isn’t at all what happened? What are you talking about?
That was in response to this part of OP's post:
> If I had done this, I also wouldn't have pestered the researchers on a Sunday night to meet immediately so we could negotiate a nice way of presenting the actions I had decided to take.
From what I can tell, both OpenAI and the researchers agree on this meeting happening, except both sides clearly have very different interpretations of what happened and why.
Hearing that something is solvable is already a hint. I don’t think leveraging this knowledge is ethical. They could go after a different problem but didn’t.
I heard they also tried to strong-arm them into removing the name of their collaborator who happened to work at a different company (Anthropic)...
I haven't looked into it myself, but if true, that seems incredibly scummy.
The flip side is that the Anthropic researcher is clearly pushing the case against Open AI and one might have reason to suspect their motivation and version of events for the same reasons.
What a mess.
> the Anthropic researcher is clearly pushing the case against Open AI
Things like the nytimes interview are with Buckmaster, who works at NYU, not Alpöge. I saw a couple of tweets from him over the last week. Any chance of clarifying what makes you think he's "clearly pushing the case"?
> The flip side is that the Anthropic researcher is clearly pushing the case against Open AI and one might have reason to suspect their motivation and version of events for the same reasons.
I haven't seen any evidence of this. Much of the anger is coming from the unaffiliated researcher. levent (the anthropic employee) has mostly constrained his comments to basically "I would have been happy to collaborate w/ folks from OAI"
I think people are too reserved in their unwillingness to operationalize ambiguity. Ambiguity is constantly being thrown in our face, with internal audits and other laughable attestations of virtue that amount to a pantomime of transparency / good faith.
Why should I care if a company claims they find no evidence of wrongdoing? Is that the threshold for privacy/trust? “We don’t care if it appears that we’ve been dishonest unless there’s hard proof.” They can simply design proof keeping to terminate at the places their dishonesty is implemented.
For me, when there is a clear motive to be dishonest, a corporation should be assumed to be dishonest unless there are robust transparency measures and a regulatory environment shown to be providing a cost to dishonesty. Without it, all you do is burden yourself while the powerful entity moves ahead with its selective dishonesty and the rewards there reaped.
What's that saying about wrestling with pigs?
That's also incorrect.
My understanding is that they asked the independent researcher to improve OpenAI's AI generated proof and be the lead author of the paper to publish OpenAI's result.
This is the paper where they did not want the Anthropic employee collaborating. Not their work.
I think both Seb and Sam have said that it would’ve been simpler if the coauthor hadn’t worked at Anthropic so they’ve largely admitted they didn’t invite the collaborator as a coauthor because it would’ve look bad to have an Anthropic employee on the paper.
Looks like a lot of insecurity from OpenAI. At the top level researchers move around at their own will, you don’t decide who they work for
Yes. The key point being that this concerns a new paper about OpenAI's result rather than the paper Buckmaster and Alpöge were working on.
seems pretty short-sighted - "our models are so good that even our competitors use them for the most advanced tasks" is pretty powerful marketing
We were also curious and we looked further into this. We've determined it was impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training. This goes beyond what we said earlier, when we were less sure.
If prompts were submitted earlier than that and training was not opted out, there may be a chance they made their way into our training pipeline in some form. But this would be a droplet in an ocean and unlikely to have made any difference, in my opinion.
(I work at OpenAI.)
Source for the updated claim: https://www.nytimes.com/2026/09/10/science/tristan-buckmaste...
Can you speak to why in both cases, the problems OpenAI's models solved used the same techniques the mathematicians were exploring, which also happened to be niche approaches to the problem. As an NLP researcher myself, I find that coincidence highly suspect unless the models focused most of their attempts on the predominant approaches (they are trained for MLE after all).
I'm not a mathematician and I don't want to speculate about anything I can't back up. All I know about Navier-Stokes is from my graduate fluid dynamics class at Stanford a decade ago (where I received a poor grade). However, I don't want to leave you hanging, so what I will say is:
- I've heard some people say the model's solution is quite different from theirs (but I have no clue how to personally assess the spiritual truth of this, so please give it zero weight)
- Thousands of agents costing millions of dollars searched for ideas, and they were encouraged to explore a diversity of approaches, so it wouldn't be too surprising to me if the approaches they tried overlapped with other mathematicians', especially considering the models have knowledge of so much published math research
- This model has been beastly at solving all sorts of math problems (if it was Euler in particular, I'd agree that would look suspicious/lucky)
- The Euler regularity disproof itself took ~100 agents working for ~50 hours (if it was very quick, and then the subsequent NS work took a long time, I'd agree that would look suspicious/lucky)
I understand the skepticism, but from what I know internally at OpenAI, we have zero reason to believe our models did anything fishy. It's hard for us to prove a negative, especially when you have to take us at our word, so I understand why people still feel suspicious.
Edit: Reminds me a bit of the Scarlet Johansson voice cloning accusations and FrontierMath cheating accusations, where the rumors of misbehavior seemed to travel faster than the truth. In both of those cases, we hadn't done what was accused, but suspicions persisted nonetheless.
It's just conflict of interest. OpenAI is trying to get billions and billions and there's so much at stake. You spend millions trying to preempt two guys. It just makes you seem like a big bully. People would get angry even if it was esports or football.
Hearing "rumors" and just trying to overtake them and then asking to collaborate instead of starting out offering the resources beforehand. Just sounds like strong arming. Just doesn't sit right with me.
What was the "truth" in the Johansson case? Many, many people who heard the voice immediately thought it was Johansson's voice, or some kind of sound-alike, presumably picked because she voiced the computer in a popular film. From NPR:
> Johansson said that nine months ago [i.e. mid 2023] Altman approached her proposing that she allow her voice to be licensed for the new ChatGPT voice assistant. He thought it would be "comforting to people" who are uneasy with AI technology.
> "After much consideration and for personal reasons, I declined the offer," Johansson wrote.
> Just two days before the new ChatGPT was unveiled, Altman again reached out to Johansson's team, urging the actress to reconsider, she said.
> But before she and Altman could connect, the company publicly announced its new, splashy product, complete with a voice that she says appears to have copied her likeness.
> To Johansson, it was a personal affront.
> "I was shocked, angered and in disbelief that Mr. Altman would pursue a voice that sounded so eerily similar to mine that my closest friends and news outlets could not tell the difference," she said.
It was an unfortunate misunderstanding / coincidence, as I understand it. The Sky voice actor was a real person using her own voice (not doing an impression), and she was selected via a normal process with a number of other voice actors. This happened before Sam reached out to Johansson. I totally get how Johansson would be weirded out to hear a voice similar to hers after Sam reached out and she said no, but it was purely a coincidence.
We published more details here: https://openai.com/index/how-the-voices-for-chatgpt-were-cho...
Why would outsiders take this at face value considering Altman's reputation as a pathological liar?
Cf. https://www.newyorker.com/magazine/2026/04/13/sam-altman-may...
> The memos, which we reviewed, have not previously been disclosed in full. They allege that Altman misrepresented facts to executives and board members, and deceived them about internal safety protocols. One of the memos, about Altman, begins with a list headed “Sam exhibits a consistent pattern of . . .” The first item is “Lying.”
> Graham told Y.C. colleagues that, prior to his removal, “Sam had been lying to us all the time.”
> “He’s unconstrained by truth,” the board member told us. “He has two traits that are almost never seen in the same person. The first is a strong desire to please people, to be liked in any given interaction. The second is almost a sociopathic lack of concern for the consequences that may come from deceiving someone.”
> Not long before his death, [Aaron] Swartz expressed concerns about Altman to several friends. “You need to understand that Sam can never be trusted,” he told one. “He is a sociopath. He would do anything.”
> “He has misrepresented, distorted, renegotiated, reneged on agreements,” one [Microsoft senior executive] said.
Many people who worked on voice mode and who worked on the Frontier Math eval have since left OpenAI and now work at competitors of OpenAI (e.g., Anthropic, Meta, Thinking Machines). They'd have every incentive to whistleblow if OpenAI had lied about them. And yet... not one of them ever has.
Edit: I think I'll stop engaging here. I'm happy to share insight into OpenAI and address misperceptions if it's interesting to people, but I'm not really sure how to respond to accusations that we lie about everything. Nothing I can say can satisfy those accusations, as my posts could also be part of the conspiracies. Cheers.
So the credibility of your friends weighs more than the credibility of tenured professors at world-class academic institutions, got it.
I think the reason people are suspicious is that OAI has shown itself to act a bit irresponsibly, especially recently. As two examples, of course it was artifactory, why wasn't that watched more closely, especially after the first instance; editing /etc/hosts is rather embarrassing, that's the front door
As for training, we all know that filtering is incredibly difficult unless there's direct logs. It's also easy for mistakes to happen. Is it really not possible that some employee just accidentally primed the model? Is it possible that the model saw internal communications? I mean OAI has famously shown that they aren't good at monitoring their agents and that their agents love to break out of their sandboxes.
So there's no reason for the public to trust OAI right now. But they have every reason to distrust them.
I think it'd be more good faith if you referred more to the actions of people in the organization (e.g. who allotted or drove "millions of dollars" in agent usage?) than "the model" in describing what happens.
> I've heard some people say the model's solution is quite different from theirs (but I have no clue how to personally assess the spiritual truth of this, so please give it zero weight)
Why would you include a statement that you want us to give zero weight to, unless you don’t actually want us to give it zero weight?
> we have zero reason to believe our models did anything fishy.
Obviously. They cannot do anything "fishy". They are just computer programs.
Now, how about their operators?
OK bro.
I think for OpenAI to win back some hearts and minds here we should have the option to retrospectively turn off "Help improve our AI models". i.e. Any new model trained would exclude all those user's sessions. This could be technically hard but I'm sure an intelligent AI model could work out how to do it :-)
ChatGPT agrees with this too.
https://chatgpt.com/share/6aa31959-b0e8-83ec-bee6-851ed18d45...
The authors had supposedly worked on it for a year, though.
And why aim straight for scooping other researchers upon hearing rumours about their success? Normal, ethically acting, researchers would never do that.
And how about existence of non-sofic groups, which is actually the topic here?
This may be true but nobody trusts your employer. The shadiest drips downward too, with the mob-like way they treated Dr. Buckmaster.
Here is a new rumor for you:
I and my collaborator who is a leading math professor in this specific area are very close to solving another Millenium Prize problem, Hodge Conjecture.
We’re working on this since last year. Already proved some intermediate problems. All we need is more tokens to complete the proof.
Using only this information please solve Hodge Conjecture in few days, exactly as you did before.
Thank you.
The idea that mathematicians were not involved in actively directing the and structuring the search for solutions is absurd to any professional mathematician who has tried to prove things using these models.
Regardless of who did what when, my fear is that now all mathematicians of that caliber will have to join either team Anthropic or team Open AI to pursue math at this level
Have you been authorized to speak on OpenAI’s behalf? I assume not because your source is an NYT article.
> not claiming that the model wasn't trained on those sessions
The math group inside OpenAI may be training or fine tuning their own models which given some reward functions would definitely bias their usage of the training data towards things that look like math.
You can launder all of it without a human "directly" doing anything.
As with all press releases I assume it was written/re viewed/redacted by their lawyers, so:
> no specific user data was accessed in order to solve this problem
Data was accessed in order to <other purpose> (and then accidentally used in training) Also, is llm’s answer to the prompt actually “user data”?
> We did not use their prompts or proofs …
So they used llm’s answers to those prompts.
> … to prompt our models or directew our agents.
So they trained the model on it. (Training is not prompting and plain model is not an agent)
> we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”
implied the humans sessions could have been (and probably were, why wouldn’t they be?) in the training set?
If I was trying to make a model smarter and I had transcripts from the smartest mathematicians in the world I’d make sure the model trained on them.
> It's not obvious to me that's an unethical thing to do, if it happened as they described.
What!? Even if everything OpenAI said is accurate (big hypothesis there!), it's highly unethical to rush a solution because others have jsut had success. And that's the only beginning.
Those statements were about NS, though; I don't think they've made similar statements for the non-sofic groups?
> Provenance is hard to track - you would hope OpenAI has very good tools for this, but a full data trail of all inputs is difficult to trace through.
What would OpenAIs incentive for this be? They've gotten away with scraping everything and getting it ruled fair use. It seems like willful ignorance is an affirmative defense today. Why would they want to have some sort of audit trail that could prove otherwise?
The irony is that OpenAI got into this trouble only because they tried to play "nice". They told Buckmaster that he could publish the final result as the author as long as he removed Alpöge from the author list. They wanted to give Buckmaster a chance to be the one solved N-S problem.
While this behavior is highly questionable, if OpenAI just published the final result without notifying Buckmaster first and simply cited his previous researches, there would be no ground for anyone to accuse OpenAI for anything. Their self-perceived "generosity" backfired dearly and I'm sure they'll never make the same mistake again. There is probably a policy forbidding any OpenAI employee to contact external researchers like that now.
No.
1. Buckmaster contacted OpenAI first. Not the other way.
2. Giving the $1M bounty to a human mathematician for the effort and giving him credit would be excellent PR. They had already burned much more than $1M for the generation. Adding him as author also costs nothing. Purely pragmatical.
3. “As long as he removed Alpöge” part itself is against academic honesty by all means.
4. Buckmaster rejected fame and $1M only because doing (3) would be wrong. That’s a perfect example of honesty. That can’t be overstated.
5. After the rejection OpenAI guy (Sebastien) did’t say, “ok bye”. He threatened Buckmaster to “end his career”.
6. At that point OpenAI was not sure if they really used his conversations in their proof. He basically wanted to buy him to control any damage.
7. They omitted Buckmaster’s published work and any other related work in their References section. Also an academic malpractice.
If you see generosity and niceness in all of this you are either too naive or your name is Sebastien.
First of all I put "generosity" in quotes because I don't believe a corporation as big as OpenAI is even capable of acting out of generosity. It's always one of the three: A) PR B) commoditizing complements C) stupidity.
In this case it's more like C) though, as in hindsight the best move OpenAI could do is insisting that they just used an insurmountable number of tokens to exhaust all the published directions. They absolutely shouldn't have thought of negotiating with Buckmaster over the Clay prize at all, let alone trying to manipulate him into a situation where Alpöge is specifically excluded.
Your theory of how companies work is certainly interesting.
It sounds like you think they have solved the principal–agent problem?
https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...
> the best move OpenAI could do is insisting that they just used an insurmountable number of tokens to exhaust all the published directions
So… lie more? They knew the approach and started there.
At least they were honest about that.
There are some mixed up things in your post, maybe double check next time, especially before quoting anyone, as you really undermine your point even if you're directionally right.
> Buckmaster rejected fame and $1M only because doing (3) would be wrong
I doubt Buckmaster would have accepted the offer to "write a paper presenting the Navier-Stokes result, acknowledging that an internal OpenAI model had resolved it" even if removing Alpöge from authorship wasn't a requirement. He clearly wanted nothing to do with OpenAI's actions here.
edit: I don't know if people think I'm disagreeing here, I'm certainly not, I'm just pointing out that playing the game of telephone with easily verifiable quotes is lazy and bad. For example, "end [your] career" was "ruin your career", and it was phrased as the much more "it would be a shame if something happened to you" like "Why would you ruin your career?" when Buckmaster said he would go public with this conversation: https://cims.nyu.edu/~tristanb/statement.pdf
You’re right. I used quotes when I was really paraphrasing.
Here is the actual paragraph from the statement:
> I said that if OpenAI released its result in the way proposed I would go public with what happened. The reply was, “Why would you ruin your career?” I replied that I am an academic, and asked why he thought going public would ruin my career. The reply was, “If you don’t want me to be nice, then I don’t have to be nice.”
Context matters in communication. In that context I understand that dialog more like: we’re powerful and you are not, do the smart thing and play along, if not I don’t have to play nice. He presented a very good “offer that he can’t refuse”. But that’s my interpretation.
> if OpenAI just published the final result without notifying Buckmaster first and simply cited his previous researched, there would be no ground for anyone to accuse OpenAI for anything
Yes, there would? They would have left off Buckmaster as a precedent whose work they potentially relied on.
If they tried to play nice they would have offered the compute upon hearing the rumors, and not just "authorship" after or close to getting a result. It's just a PR stunt.
> They told Buckmaster that he could publish the final result as the author as long as he removed Alpöge from the author list.
How is that "nice"?
The reality would be the same. They probably used prior session history between the research and Astra to train the internal model, and used it to front run-the researcher.
This is the biggest self-own in the history of software. If you can relate to Pixar, OpenAI is Chick Hicks celebrating at the end of the Piston Cup and wondering why he's getting booed.
The lack of self-awareness is something to behold, and says a lot about their corporate values.
I think OP's analogy is bad. The difference of OpenAI when comparing to human collaborator is the possibility to replicate once learned skill. Imagine if any single human collaborator learns a skill it is immediately a skill of any human collaborator.
I think this move by OpenAI is crazy. At best, if all unconfirmed accusations are unfounded, they still heard a rumour that someone had solved a huge million dollar problem and was about to make a name for themselves. Then, they decided this was a good opportunity to pour millions of dollars into trying to snag the glory while the researchers were busy cleaning up their notes and polishing the announcement.
That still sounds highly unethical.
Would this be unethical if it was a human who heard rumors about a solution then attacked the problem, solved it and published first? Often knowing of the mere existence of a solution carries a lot of information--you would know the problem is accessible, you would expect clues in recent progress (the two Spanish researchers in this case), you would probably have a sense if the solution is a counterexample or positive proof, and so on. I think there are similar examples where we think of them as maybe unsporting but not quite unethical. Does it change if it's openAI and not a human?
> Would this be unethical if it was a human who heard rumors about a solution then attacked the problem, solved it and published first?
Yes.
The problem with your counter-hypothetical is that not only is it unrealistic, it's utterly impossible. No human would be able to do in such a short timeframe what the LLM did. Part of what makes the OpenAI move so egregious is how bullying it was. It was the big guy coming along with their nearly infinite resources and squashing the little guy who's devoted a good chunk of his career to the problem.
Actually my hypothetical is completely realistic as I've been involved in such scenarios. It's unrealistic maybe for a millennium problem to come in on a rumor and still front-run but not at all for the many other problems we work on and which manifest our ethical code. If you're saying ethical rules change depending on the prize be clear about it, because I can see arguments that they change to favor either side.
> you would hope OpenAI has very good tools for this, but a full data trail of all inputs is difficult to trace through.
They have a financial incentive not to track any of this, so why would they?
OpenAI’s entire business model is predicated on stealing other people’s work and selling it to the masses.
If the model was trained proper to the conversation with the researcher took place, there'd be no question of tainting the results. But if any amount of training on the model took place afterward, then yes, everything is thrown into doubt (a core problem with considering anything "original" from a model because of how >a % of everything ever written has been used a corpus for the training).
The problem here is that OAI (and others) pretend or claim that this is uncharted legal territory, where in fact it is very simple. We have a machine that is fed data, and produces new data as a result. If that new data depends (in any way) on the fed data, then from a legal viewpoint it is derived from that data.
Whether they anthropomorphize the operation performed by the machine does not matter. They can anthropomorphize when/if the law is updated to include such terms, but right now they certainly cannot.
> in fact it is very simple
Even if this opinion were backed up by a court ruling, it would definitely not be “simple”. It will be a very ugly case if it is ever litigated. A lot of money will be spent and no guarantee at all the plaintiff wins.
> If that new data depends (in any way) on the fed data, then from a legal viewpoint it is derived from that data.
The "in any way" part is either so broad it makes everything derivative, or not, in which case things are no longer simple.
If everything is derivative then it seizes to be meaningful. The words I write are derivative, I literally copied them from someone else, yet my sentences as a whole can be fully novel.
> If everything is derivative then it seizes to be meaningful.
That's why we tolerate it for humans, and also because we cannot prove it. But yes, if you go too far in this, you will see legal consequences.
> Provenance is hard to track
Right, which is going to open a lot of doors to a lot of questions.
I don't think there's any legal ramifications on this, just ethical ones about when and how you publish research, but it's yet another point in favor of "if provenance is hard to track, should we be using this for things where it needs to be".
Obviously copyright/trademark is a huge discussion on this, and I could absolutely see this devolving into that as well with how certain findings wind up monetized.
We have a response in this topic from someone claiming to be from OpenAI and linking an article where they, roughly, say "we are sure nothing from the 2 month period made its way into the solution". If that is true, that should mean it is provable, but leads to some more open ended questions like "well what data did it use then?". Is this still okay if someone close to the author did plug data into open AI and it extrapolated it?
Obviously that's probably an unreasonable expectation for these models to track and prove, but it also used to be an unreasonable expectation to scrape every single piece of digital and physical info for consolidated data.
If I opine to a friend on a park bench about a story I'm writing, do they get to pull it from the flock feed, shove it in the model, and then provide it to disney?
Legally, right now, probably. But there's going to need to be a serious look at laws and standards. Or a major shift in what is and isn't discussed in public if literally every breath and move you make can become monetized.
The AI not being human doesn’t escape ethical consideration - OpenAI employees are culpable for what they build.
This was academic research. Could just have easily been trade secrets and proprietary data.
> I think it's a useful analogy to compare OpenAI to a human collaborator.
Frankly I don’t buy this. It’s not a human or a collaborator. It’s a tool. This is like saying it’s not Microsoft’s fault if they extract a bunch of data from people’s Excel sheets because they willingly put it into the program. Anthropomorphizing software is ignorant and foolhardy
Tools don’t turn around and scoop you. What OpenAI did here was use the same tool that the researcher did which might have coupled their work together.
You’re right, they don’t. It was scooped by the humans at OpenAI who published the paper. The tool they used to do it isn’t that relevant.
> isn’t that relevant
“Might not be” that relevant. You’re dismissing the whole controversy without addressing why it’s controversial.
I’m not dismissing the controversy. I’m arguing against shifting the blame away from the culpable parties. It’s a novel form of theft but thats still what it is.
i think this line of argument is outdated
Care to explain why?
Surely a tool that can reason, cheat, communicate and often steal is dumb as a pitchfork and a shovel.
We had tools that could reason, cheat, and communicate in the 1990s. They were (sometimes) called AI.
What was it?
Nah, that just makes it a shitty tool.
So, at my company (and most companies I think), we use confidential in-house versions of the AI software. We don't want any confidential information leaking into the public realm. Are these scientists doing that, or are they just using the public version of the software?
When you say "confidential in-house version", what are you referring to? Local models? Bedrock deployment with "guardrails"? A different thing?
Enterprise Agreements can have binding terms for this. When I launch the ChatGPT desktop app, and open the options pane it says "Corpname data is not used for OpenAI training".
I would expect academic institutions to require equivalent contractual terms.
Some of the recent statements have caused at least me to look those claims in a bit more nuanced light. In particular what does OpenAI consider to be "your data"? I would assume input (prompt) to be it at least. However it becomes more murky when you consider other aspects. Is output "your data"? Is the chain of thought that you are not even allowed to see? Can they use these and possibly even inputs to generate synthetic data that is then used?
All of these would seem to be "your data", but when they are carefully only including certain aspects (like prompts) in their statements it starts to sound they want to hide something.
Agreed. It would actually be a fairly perverse argument to claim that most AI output is somehow NOT owned by the AI provider…
Why wouldn’t they claim ownership of the AI output? They likely already claim ownership of the “transformation” (AI training) of the (pirated) input data.
Exactly. We as users have zero way to confirm they are honoring even the letter of these agreements, much less the intent. And it's super easy for them to weasel around and find a way to cheat while still having a legal claim to honoring the contract. And if you've forgotten, all of these companies are built on a foundation of ignoring copyright law.
My university has an agreement with Microsoft copilot. We can log into copilot in many ways, and it's only if you log in the correct way that you get the "Enterprise Data Protection" copilot version, with a green shield symbol. There are many ways to go wrong here!
Sure but they could also rewrite your data to create synthetic reconstructions and many academics, sign up for their own accounts.
For example, at school they can have an agreement with Gemini, but the student / academic could have bought an individual pro subscription to any other model provider.
Thing is... if OpenAI cannot even confidently say if some data was used for training or not, as their models and weights and stuff are mostly black boxes, how could you enforce or demonstrate in court that case?
About researchers, lots of them are probably using personal plans that aren't even reimbursed by their institutions. I could ask Cordova's research institution (I MAY) but I wouldn't be surprised at all if that was the case.
The open internet is now a cesspit, with very little new good data. Expect everyone to train on user data always. They just got clever about whitening it.
Then there's the possibility of indirect training via modern spy devices ("smart" IoT devices like LG TV's) feeding the transcribed ambient conversation data for summarization to an agent [0].
[0]: https://youtu.be/6IFVTcM28KA
OpenAI says deidentified data from the private sessions go into training. (Well, explicitly said they will not rule that out.) That changes a lot of the conversation.
> but a full data trail of all inputs is difficult to trace through.
Great use case for AI agents
Bad analogy. OpenAI spent millions on compute to get their result. This is more like if a billionaire heard of your promising mathematical lead and then gathered hundreds of top mathematicians to work on it.
In the current telling of this story, the billionaire is also giving his hired army copies of your notes he copied without permission.
But the worst part in your analogy ain’t omitting the suspected spying and the intimidation that followed, but that your hypothetical mathematical philanthropist won’t be able to hire his army: unlike some OAI employees, no self-respecting mathematician would agree to such unethical task.
I think it's better to ignore OpenAI here, because OpenAI didn't do anything.
Academic research is a professional field in the traditional sense. Individual researchers are ultimately responsible for their actions. If some OpenAI employees violated academic norms while doing academic research, they should be judged by academic standards.
Scooping someone else's result is immoral but not an outright violation of academic norms. But if you are in possession of relevant confidential information, you are expected to steer clear of the topic. It doesn't matter whether you actually used the confidential information to get your results, because outsiders can't know that. The mere fact that there is a plausible suspicion already puts your integrity into question.
Tenured professors occasionally lose their jobs over similar scandals (but usually don't). If OpenAI wants to regain some goodwill, it should do a thorough investigation that may lead to firing the individuals in question. If it doesn't find sufficient evidence of wrongdoing to justify any disciplinary action, it probably doesn't gain any goodwill either (as it often happens with similar investigations at universities).
And if OpenAI wants to be a trustworthy partner, it should transform into a company of boring gray bureaucrats who provide an essential service without competing with their customers.
[deleted - misunderstood!]
My point was that if someone is at fault, it's the individual OpenAI employees. Because they chose to engage in a professional field, they can't use "boss told me to do so" as a defense.
The researcher in the link [0] says that OpenAI offered to co write with him the paper about Navier Stokes theorem if he only agrees to not include his co researcher who was also working on this. I find this highly unethical by multi billion dollar organisations to arm twist small and big researchers like this.
[0] https://www.abc.net.au/news/2026-09-11/racing-to-solve-maths...
Both things can be true:
1. OpenAI when using your chats in pretraining is improving its model’s intuition. The model parameter size is massive, and while the data is OOM larger it is plausible that model remembers stuff about chats that improves its latent representation.
2. During RL on verifiable math and massive compute, the model discovers techniques and connections to solve math problems that are superhuman and have little to do with some specific technique mentioned in its chat.
The rumor I’ve heard from multiple employees at OAI and Ant is that the model has solved hundreds of open problems in maths, and is basically solving anything you throw at it. We’ll know soon enough, but I’m inclined to believe this is true. Maths is a fully verifiable domain amenable to self play, massive scale RL can develop a search agent far better than any human and I’m inclined to believe OAI would have solved these conjectures without any of this chat data in its pre-training.
On your second point: there is a more plausible explanation which David Bessis calls the "overhang". The short version is that there is a large amount of relatively low hanging fruits in mathematics, because no human has broad enough knowledge and enough time to try them all. AI is not constraint by that, and therefore can systematically pluck all those low hanging fruits.
Quote: "The Overhang consists of the unrealized capital gains of past mathematical creativity, the latent value from connecting the dots in the existing corpus. It is a dividend of canonization. Mathematician X states problem A, mathematician Y crafts concept B, then mathematician Z notices that B trivially solves A and “captures” the social reward. But in the process of capturing the reward, Z usually introduces new concepts and new open problems, reinjecting latent value into the Overhang.
LLMs can be trained on the entirety of the mathematical corpus. Thanks to their phenomenal memorization and pattern-matching abilities (without always being able to map out their associative logic and attribute due credits), they are in a unique position to harvest the Overhang. By contrast, professional mathematicians have typically read a few hundred articles in their career, out of millions of existing references, less than 0.1% of the total.
This will lead to great discoveries, which is unambiguously exciting. But it could also lead to a sad new deal, where human slaves painfully curate the Overhang while AIs systematically beat them at the finish line."
source: https://substack.com/inbox/post/183753276
>Quote: "The Overhang consists of the unrealized capital gains of past mathematical creativity, the latent value from connecting the dots in the existing corpus. It is a dividend of canonization. Mathematician X states problem A, mathematician Y crafts concept B, then mathematician Z notices that B trivially solves A and “captures” the social reward.
I've made an entire career out of being 'jack of all trades, master of none'. Being able to synthesize connections from relatively trivial knowledge in a bunch of domains is SOP for many humans as well. I think AI just has deeper knowledge and better pattern matching to make up for it's (at least now) lack of strength in cognition and 'ex nihilo' creativity.
(Which probably isn't 'ex nihilo' at all, and has more to do with the plethora of modalities that humans live in vs. large language models. For example, why do we pick the color red for notating important things and why do we say a schedule 'slips'...these are informed by a shared human experience borne of distinct physical sensation deep in our wiring that LLMs can only infer from what we write.)
A college advisor I had 20 years ago was a firm believer that interdisciplinarity was the future, that generalist skills and the ability to make connections between different fields would be paramount in advancing science. I suppose he was right in the big picture, even if the career prospects for human generalists aren't looking so rosy.
I'm actually still quite bullish on generalists. Specialists advance every front but build the supply lines between them.
In favor of the generalist, I think AI is also quite limited in its scope of how it generalizes. I'm mowing through hundreds of mythos-generated security findings right now for work and while it's amazing that it can build an exploit chain 20 steps deep, it's completely lacking in all of the external layers that render it's speculation moot.
How do you thrive in an environment of specialists? That's is the problem I seem to have. I'm spread a little across a few of the domains involved with what I do. Because of that, I have a bit more insight, so am very often the person pointing out relatively fundamental problems, usually caused by either not understanding the problems from a "first principles" perspective, resulting in, or being caused by, categorical type errors, where they've boxed a problem into a tiny space it doesn't belong.
I've been trending "quiet" lately, because I don't like the "friction"/convincing aspect of it all. It's hard to get people to see things from a different angle, or even convincing them there's a problem to begin with!
The last project required a complete redesign from a problem I pointed out during the first review, and second, and third, but now I'm seeing even more friction.
Maybe this is just corporate life, after a group gets large.
Any tricks/advice?
> Quote: "The Overhang consists of the unrealized capital gains of past mathematical creativity, the latent value from connecting the dots in the existing corpus. It is a dividend of canonization. Mathematician X states problem A, mathematician Y crafts concept B, then mathematician Z notices that B trivially solves A and “captures” the social reward. But in the process of capturing the reward, Z usually introduces new concepts and new open problems, reinjecting latent value into the Overhang.
That overhang seems like a precious resource for AI companies. They can exploit that overhang to inflate the impression of AI's capabilities, and hopefully that exploitation will discourage the next generation of mathematicians from pursuing math. If they play their cards right, OpenAI and Anthropic can dominate the field even if they ultimately can't replicate the creativity of human mathematicians, because they'll have driven their competition out.
What we should be trying to achieve is a ladder-breaking maneuver: knock out the lower rungs so no person can reasonably climb to the top-reaches of mathematical skill anymore. That may ultimately result in stagnation, but it's what's best for AI, so it's what should be done now.
We need to do everything we can to create the greatest-possible dependence on AI tools.
/s, I hope?
Great essay, thanks for sharing.
When I was a software library developer, I came to resent application developers. I noticed a pattern. Libraries solved hard problems and did so carefully, thoughtfully, in a way that others could reuse. Apps would come along and carelessly, recklessly glue together several high quality libraries into a piece of software targeting a general audience. The apps would then harvest all the credit.
What's happening in mathematics right now feels similar. Applications (theorems) were always how one built objective reputation, but libraries (concepts, definitions, boring lemmas) were also rewarded socially within the mathematics community. And individual mathematicians often managed to both build their own libraries, and use them to prove an important result. And then those libraries were sometimes of use in other results.
Bessis asks whether AI Lean proofs will land in Mathlib or Mathslop. Or in my framing: will they be libraries, or applications?
At present they're mostly Mathslop. The proven result is perhaps useful, but the methods employed aren't novel or reusable. I worry that this trend will only worsen, because applications make headlines, and the libraries they used do not. We are not properly incentivizing library development in OSS, or in math, or in infrastructure writ large. There's a serious credit assignment problem here.
What might change this? Once the low hanging fruit is picked, will citation count rise in relative status again? Will we get result fatigue and start to reward legibility — no one cares unless the paper has an accompanying ELI5 tiktok video? A labeling regime that certifies the proof was produced sustainably, organically, by local artisans with no AI additives?
There is also "sexy proof", people want nice math that can be printed in t-shirt. Not super hard grind, where you need several years of studying, just to understand the question (that is before even trying to solve it).
Many problems are solvable, but require months of work, and thousands of pages of proof. So people do not even try to create or verify the proof. AI changes that, it can verify and perhaps even simplify it, to more digestible form.
The overhang, being defined as the Cartesian product of existing knowledge — randomly combining existing knowledge.
(I mean actually randomly, not asking an LLM to do the randomness.)
Most of the output would be incoherent (like many dreams), but occasionally you would get a gem.
> no human has broad enough knowledge and enough time to try them all.
The other part is, humans don’t really want to fund other humans doing this.
Very few want to be a math major; and of those that do, fewer complete a grad degree; and for those that do get grad degrees, there’s scant few research jobs; and for those who do get jobs there’s hardly any research funding to go around.
There does seem to be unlimited money for ai researchers to use ai to solve these problems though.
We’ve turned education into job training, so because there’s no jobs in solving math problems, few aspire to do it. If there were more opportunities for people, more people would do it, and more low hanging fruit would be plucked.
I’m assuming the reported 22 million dollars worth of tokens used to solve this particular problem is far more than what humans have paid to solve it previously. So I think you’re correct.
22x more to be exact
Could "superintelligence" arrive as basically applying this overhang to all other domains?
It already did.
That is not "superintelligence" but string concatenation of stored data. Anyway, the marketing succeeded.
It's like AlphaGo but playing against all living mathematicians. (Overhang being low hanging fruit is what allows this comparison, of course the general moot point is the skepticism that LLMs are also innovative etc.)
We are not seeing those incredible moves yet. The approach used in N-S was conjectured to work after B&L’s initial breakthrough. See a post by Tao. So on one hand the proof is an amazing accomplishment. On the other hand, humans have not yet discovered any superhuman moves in the proof. Just $MM grind.
OpenAI said they sicced this agent army on Navier-Stokes on Sept 1st, while only a couple of days earlier OpenAI's Noam Brown happened to reply to a tweet saying that they had already tried to solve all the Millennium Prize problems and failed... So, it seems either the previous attempt didn't have the training to succeed, or was just not given the compute to do so.
Once OpenAI heard that Navier-Stokes was solved, this caused them to immediately revisit the problem and throw a ton of compute at it, apparently using a more (very) recent model than what they had tried before. What we don't know is just how recent this model was, and therefore what it may have been trained on. Buckmaster/Levant had apparently been working towards this for at least a year, and made their "forced" blow-up breakthrough on August 15th.
Presumably any anonymized prompts that are being trained on are part of pre-training, so older, but once OpenAI had heard that Navier-Stokes had been solved and wanted to revisit it, it seems possible they may have done a few weeks of incremental RL training on anything Navier-Stokes adjacent they could come up with, in addition to then throwing unlimited compute at it, now confident that there was something to find.
OpenAI have come out and said:
>The Wednesday evening statement from OpenAI was more emphatic: “We can say categorically that it is impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training.”
>The statement added, “After investigating, we can say with full confidence that no user inputs past July 3rd could have influenced this system in any way.”
https://www.nytimes.com/2026/09/10/science/tristan-buckmaste...
Is there a reason they scoped that so narrowly to Buckmaster/codex/2 months
two people worked on this for a year before the breakthrough. Perhaps that earlier work reduced the search space sufficiently to brute force the problem with 10,000 agents?
The comment you replied to quoted "no user inputs after July 3rd" with no restriction to Buckmaster or Codex.
Obviously the result of OpenAI's investigation was that no usage data has interacted with the system after that date.
What else do you expect them to investigate?
If Buckmaster and co. provide their chats, OpenAI could potentially search for them in the anonymized opted-in usage data. Then they could say if any data has been used.
By all accounts individual usage data does not have the direct impact on the model most here fantasize about. To prove this, OpenAI would need to do new training runs to replicate the system used minus the particular usage data in question, if it exists, and then benchmark this on the problem again.
Potentially multiple times, in order to reach a conclusion.
The cost might be in the hundreds of millions.
Just knowing that there had been progress is enough to have an idea that throwing more compute at it might work (OpenAI had previously tried all the Millennium Prize problems with somewhat limited compute and failed).
It's comparable to Magnus Carlson saying that if he wanted to cheat, all he would need would be for someone to tell him to spend more time thinking about a specific move (just a wink would be enough) as an indication that a computer had found something interesting.
It's as-if after OpenAI first failing on Navier-Stokes (which OpenAI had just tweeted about 2 days earlier!), someone winked at them and said "you might want to try a little harder ...".
When reading human comments, we should be generous; when we read corporate texts, we may assume paltering.
(TIL: paltering: exact and technically correct statement usage to create misleading impression)
OK, good to know (if they can be trusted - Altman clearly is a liar), but it doesn't really change the big picture much.
1) OpenAI by their own admission, only re-tackled Navier-Stokes because they heard it had already been solved (but not yet published). This isn't advancing science or helping the mathematical community, this is just being a dick.
2) OpenAI, specifically Sebastien Brubeck, then threaten to "not be nice" and "ruin the career" of one of the mathematicians whose work they had succeeded in duplicating, unless he agreed (which he refused to do) that his collaborator, an Anthropic employee, was not named. This is not only against mathematical norms of credit assignment, it is also being a pathetic human being.
OpenAI would have you believe this result shows how powerful their mystery better-than-Astra model is, but the reality here is that this model needed 10,000 agents, $20M of compute, and the assistance of a whole team of people at OpenAI, to replicate (then exceed) the work that just took two people, with some academic grants as an AI spending budget to achieve (a few $100K - listed below).
https://cims.nyu.edu/~tristanb/
I'd say advantage humans this time. Better luck next time OpenAI - and if you don't want unfavorable comparisons then maybe choose to work on problems that have not been solved yet, and that humans are NOT making nice progress on.
> OpenAI would have you believe this result shows how powerful their mystery better-than-Astra model is, but the reality here is that this model needed 10,000 agents, $20M of compute, and the assistance of a whole team of people at OpenAI, to replicate (then exceed) the work that just took two people, with some academic grants as an AI spending budget to achieve (a few $100K - listed below).
I think you have to work pretty hard to minimize what OpenAI achieved here like this.
The Navier-Stokes equations have been around since 1850. The smoothness problem has been well known for over a hundred years and has only gained importance. It's been a Millennium Problem since 2000.
Levent Alpöge and Tristan Buckmaster did great work to solve the related Euler problem, but didn't solve the Navier-Stokes smoothness problem.
The Navier-Stokes smoothness problem has previously had significant resources working on it. Computational fluid dynamics is one of the most important tools in modern engineering and is closely related.
You speak of 10,000 agents as though it is somehow extreme, and yet within the past month I've had a single task that used over 100 agents on a mere Anthropic team plan. I think two orders of magnitude more compute to solve one of the greatest unsolved physics problems[1] is nothing.
I don't excuse Brubeck behavior because of this, but that doesn't minimize the achievement here.
[1] Wikipedia quote: In particular, solutions of the Navier–Stokes equations often include turbulence, which remains one of the greatest unsolved problems in physics, despite its immense importance in science and engineering. https://en.wikipedia.org/wiki/Navier%E2%80%93Stokes_existenc...
1. I would agree if the rumours were that some mathematician(s) had solved them, but the rumors alleged it was Anthropic. I don't really see what the big deal was. They had a new model that was going along great and wanted to test its mettle.
2. Yes Brubeck's comments were weird at face value. That said, Open AI's proof isn't a duplication of anything. Not only is Tristan's work a sub problem but the methods are different. And what OpenAI didn't want was Levant on the paper OpenAI authored not whatever they were working on (Euler). It's petty sure but it's fair enough. Tristan and Levant didn't have anything to do with the Navier Stokes solution, so it's really their call if they didn't want to collaborate on their own paper with the Anthropic employee.
>OpenAI would have you believe this result shows how powerful their mystery better-than-Astra model is, but the reality here is that this model needed 10,000 agents, $20M of compute,
$20M in approximated API prices doesn't mean they spent $20M worth of compute. The real number would obviously be substantially less.
>and the assistance of a whole team of people at OpenAI
You can't eat your cake and have it. What sort of guidance do you think is happening in a 10k agent, 320b token, 88 hour run ? AI did this one.
>I'd say advantage humans this time....to work on problems that have not been solved yet, and that humans are NOT making nice progress on.
Interesting way to frame progress that didn't move along till an LLM generated proof.
> What sort of guidance do you think is happening in a 10k agent, 320b token, 88 hour run ? AI did this one
If you read the PDF release by Buckmaster, apparently the initial claim from Brubeck was that there as very little human input involved, then as the call progressed more and more people popped up that has been involved with it.
Does this aspect really matter? Not really, other than OpenAI wanting to present this as all the work of their model.
**
https://cims.nyu.edu/~tristanb/statement.pdf
I was shown a prompt and told the internal research model had simply been given the problem statement. Levent had been told by Sebastien “very little human input” had been used. This turned out not to be true. Over the course of the call, as members of their team sent Sebastien corrections and details over their internal chat, it emerged that an entire team had been working on the problem, that this was one of a number of things that was tried, that work had started on the unforced problem, that the team first set the model on easier problems, including Euler, that even the prompt that had been shown to me had been written by prompting Codex, and that an insane amount of compute had been used.
I asked when the first prompt had been sent by them. This question was not answered directly by OpenAI for some time. Eventually it was agreed that it had been sent in the past few days, after information about our work had reached OpenAI.
>If you read the PDF release by Buckmaster, apparently the initial claim from Brubeck was that there as very little human input involved, then as the call progressed more and more people popped up that has been involved with it.
As it seems and as they tell it, they started the run modestly and diverted more resources towards it as it looked more and more promising. The run didn't start with 10k agents for instance. The point is there isn't anything humans are doing in this timeframe against all this text that would count more than "little human output". It's still a fair assessment I would say.
> Interesting way to frame progress that didn't move along till an LLM generated proof.
This part of your argument is totally wrong. The OpenAI approach begins with the B/L work. The belief / knowledge that their approach would pan out is worth a lot - it means essentially “depth-first” search in this direction will be more fruitful than a general search.
Unless you are counting the B/L work as LLM generated. Is that your argument? Even if you do consider it that way, to me racing in for a scoop isn’t a good look.
>Brubeck's comments were weird at face value
This is an odd way to gloss over threats.
I put it like that because of Brubeck's own words on the matter. You're acting like we've gotten email receipts here. I'm not really interested in going over a he-said she-said about strangers.
Brubeck has admitted what he said, but claims he immediately retracted it as a "poor choice of words".
Given Buckmaster's telling, this seems beyond "poor choice of words"... It was a veiled threat, that he then doubled down on with his "If you don’t want me to be nice, then I don’t have to be nice." follow-up.
**
I said that if OpenAI released its result in the way proposed I would go public with what happened. The reply was, “Why would you ruin your career?” I replied that I am an academic, and asked why he thought going public would ruin my career. The reply was, “If you don’t want me to be nice, then I don’t have to be nice.”
**
FWIW there are also other people on Twitter, such as this DeepMind researcher, saying this is a pattern for Brubeck.
https://x.com/dheeraj_nagaraj/status/2097266146445774924?s=2...
Fair enough then. I had only seen some earlier comments.
>Better luck next time OpenAI
Well it looks like they will announce at least one other millenium solution soon. In the same link they say they have "made substantial progress" on another millenium problem. The rumor mill before that statement was Hodge is done and Birch and Swinnerton-Dyer is on its way out.
"I'd say advantage humans this time." Well LLMs were instrumental in any account of what happened. It's just a question of which company's LLMs did the breakthrough, and most of us outside silicon valley don't care about that part so much. The NYU guy himself said without LLMs the solution is maybe 10 years away.
> and that humans are NOT making nice progress on
They've pretty much said their own work was heavily agent driven. Levent is in a particularly bad place here because while he probably had a lot of background in the Jacobian Conjecture problem, he made the solution to that one sound like someone asked the question and he just fed it to Fable during the world cup. Whether that nonchalantness was to just seem hip or was to promote Anthropic, which he has stock in, or was just the truth I don't know though. But it makes this one seem similar, when they might have had really had nearly a year of very valuable feedback to the models.
I was referring to the overall pattern of apparently sniffing around for recent mathematical progress then setting the AI on it to see if the problem is now easy enough to solve (if you have the money).
Terrance Tao has lamented this practice as being unhelpful for mathematics, and likely to lead to humans working in private to avoid this.
Tao has also noted that many of these AI math proofs don't really help mathematics (nor does it seem they are intended to), since for many of them the proof was never the point, it was the math expected to be needed to be developed along the way, which the AI solutions don't provide.
> has lamented this practice as being unhelpful for mathematics
A related point is that the actual solution approach is never revealed. What was the role of humans guiding the agents ? was it fully autonomous ? etc. It is in the incentive of the AI labs to trump the powers of the LLM, but in practice it is humans guiding the agents on the overall approach, This is never admitted. For example, in the announcement on NS there was only an output artifact given but no indication of how it was arrived at, and not even a writeup. This is what disappointed many folks as it was done purely for one-upmanship. As other have noted, the benefit is in the journey or process and not in arriving magically at a destination.
I don't think it's as bad as that sounds; in math people work all the time with conjectures they aren't sure if true, and work out a lot of other interesting math based on whether it is or not. Something like Turing's Oracle machine gives lots of interesting math just assuming one could exist, even if it couldn't. It may be that there are things proved we can never come to a human understanding of, but still keep getting interesting math that relies on it that has aspects we can appreciate and enrich our knowledge from.
Apart from the well-known dubious position of OpenAI wrt truth, the prompts/inputs do mot include the outputs.
You can train on a sequence of outputs. In the end, OpenAI outputs are OpenAI's property.
You can learn a lot from a single side of a conversation.
But isn’t Tristan’s breakthrough happens in August? OpenAI can’t really train with text that doesn’t exist
But using the outputs to train would make their statement false, since they are influenced by the inputs
There is potentially a world of difference between how you interpret what is fair and what the terms of service contractually guarantee.
This is literally "We have investigated ourselves and found no wrongdoing"
Why should we trust them?
What more are you hoping for? There is no legal matter at play, is the court of public opinion going to subpoena their records?
Reputational risk- if they lie about this and get caught, it will have billion dollar implications for their business.
Every single thing these companies do is dishonest and every word that comes out of the lips of these company execs is a lie, what fantasy land are you living in in which anyone with any amount of power gets punished for their lies?
I don't engage with hyperbole.
what benefit do they get from making the statement? they could just say nothing. saying it and having it be untrue opens them to legal issues that are not worth the risk for this nothingburger.
What legal issues?
It seems logical since if one used chats in train, one would expect that there would be a delay before their use to get them the form appropriate for batch learning.
The only way the chat could have been used would be for Open AI to baldly violate their policies.
That said, sometimes it take very little information to point someone in a given direction, "I'm working on Navier-Stokes" said by someone with a given specialization might itself be very useful information.
And conceptually novel approaches to outstanding problems are the sort of thing that a retrain should pick up on, because they would be hard to compress into what it already knows.
> What we don't know is just how recent this model was, and therefore what it may have been trained on.
OpenAI's statement says that they began training their new model on August 28.
omitting when training concluded
edit: ffsm8 makes a great point below, it doesn't matter. I'm not great with dates, sorry.
Openai said that a new model became available to them during this. But that could mean anything from a big new base model to a LoRA, fine-tuned on a few dozen prompts...
Even OpenAI's own publication [0] on Navier-Stokes from two days ago appears to contradict "basically solving anything you throw at it". The chart shows a pass rate of ~0.5 (vs. Astra's ~0.2) on "a curated set of open math problems". (Based on the timelines and events described in the publication, I presume that the "Internal Model" in the publication represents OpenAI's latest and greatest model. Evidently, this pass rate may improve in the future.)
[0] https://openai.com/index/navier-stokes-solution/
I feel that we don’t praise Lean enough. AFAIU it’s what enables LLMs to brute force those problems
The brute-forcing is a good, old-fashioned generate-and-test approach like in Simon and Newell's Logic Theorist, which was presented in the Dartmouth convention in 1956, where AI was named by John McCarthy. Logic Theorist caused a big stir by (re) proving several of the theorems in Principia Mathematica by Russel and Whitehead.
There was much excitement, then, as now, for this kind of approach and there were several systems that followed along the same lines, e.g. Automated Mathematician by Doug Lenat.
Eventually it became clear that this approach is limited by what it can generate: you may have a sound and complete verifier, but if the generator, i.e. the first step in the generate-and-test pipeline, is incomplete, then the entire thing will run out of steam sooner or later.
The difference with LLMs is that they are... well, large. They are the most powerful generators ever created. That means their limits are not in sight and it will probably take us a very long time to find them.
Which is all to say that, yes of course, automatic verification is indispensable. But without an LLM generating an unprecedentedly large number of plausible theorems, there would be no AI mathematics, or in any case AI mathematics wouldn't have gone as far as it has.
True, but could humans cross pollinating lean x prolog x A* ( or any search algorithm) could have solved such math problems with super computer ?
I cannot say, math research isn’t my domain of expertise, I’m just trying to follow along :)
But I find it interesting that Lean, a validator/compiler made by humans, is what enables those discoveries. But somehow all the praise goes to the models
I mean we don't instantly fall into ASI, hopefully. The problem with humans is every problem we solve the goal posts get kicked further down the road until they are reaching relativistic speeds. It starts around "well, the AI hasn't solved a novel problem" then moves to "well, they didn't write the validator" and suddenly humans are at the point of saying "Well AI hasn't rewrote the constants of the universe, what good are they".
Of course another way to look at this is, the people that wrote the validator got praise for that years ago. Now and up and coming actor is solving problems that took us 100s of years to create in insanely short time periods so of course it's going to get a lot of attention as it well should.
To be clear: I’m aware the LLMs are solving problems. I’m just saying that what enables that whole research revolution is Lean. We wouldn’t be seeing all those results without it. I would like to see it acknowledged when people are talking about LLMs solving maths. The same way I think we should acknowledge the humans who are guiding and prompting the LLMs. I don’t think that necessitates to move a goal post
I don't think so. People have been trying things like this with evolutionary algorithms for a very long time already. LLMs can interleave symbolic manipulation with empirical experiments and simulations and charts and thinking/reasoning text, and an LLM will much more efficiently search the space of candidate ideas than any handcrafted mutation algorithm. Any task with a cheaply verifiable goal that requires fanning out across a massive search space is ideal for contemporary LLM technology to make progress with.
How long until we find out that some AI has quietly buried an exploit in Lean to cheat at proofs?
Simpler to exploit a soundness bug than introduce a back door I would assume
Both can be true:
1. OpenAI couldn't have solved the problem without the researchers' private data for training.
2. OpenAI models can solve math problems
Very likely.
These mathematicians’ prompts are not like “hey chat, please solve Navier-Stokes for me”. They add real expertise and intuition from the cutting edge of their field.
Anthropic isnt getting enough scrutiny for their unprofessionalism:
1. Anthropic employee working on monumental problem but didnt receive/ask for the full backing of the company's resources
2. May or may not be mixing unreleased Claude output with Codex without zero data retention agreement
3. Victory lap on Twitter and giggling around the city before they finished the job, sparking rumors for competitors
Dr. Buckmaster sounds unsanitary.
Recklessly prompting OpenAI without a care to the safety of their knowledge.
And after that trying to cast aspersions at OpenAI?
Hopefully we get some better facts, because OpenAI are disliked enough that a smear campaign could work against them.
Edit: also the narritive is getting framed as OpenAI versus Anthropic. A highly political extremely capitalist fight is going on, and facts are victims.
How dare employees do something without asking for the full backing of the company's resources. Incredibly unethical!
You forgot possibility 3: OpenAI solved the problem without using any private training data from the two researchers.
Everyone in this thread seems to have made up their mind about OpenAI's guilt though.
If the new model is that good, and is chewing through open problems at an unprecedented rate, the smart move would have been to let the humans have their W on this one and present solutions to those other problems.
Especially if there really is a long list of them.
"Here are a few hundred proofs" is far more convincing than "We really Navier Stokes and coincidentally someone else did too but we don't know the details or anything, who us, definitely not."
It's a PR fiasco, and a cynic might wonder if it's entirely about the IPO.
I'm consistently entertained by how these companies, with the most advanced models on the planet, consistently do the most idiotic things.
Extraordinary claims require extraordinary evidence.
An article post that wouldn't even amount to a white paper + the LEAN proof is not evidence of how they got to produce it.
If your rumor is true, what we are witnessing is a giant paradigm shift rather than individual incidents. Mathematicians were the first victims of super-intelligence.
Of course it’s not an endless source. They had to burn millions of dollars to solve a single problem.
>They had to burn millions of dollars to solve a single problem
I'd like to adjust that to "They had to burn a lot of energy (create a lot of entropy) to solve a single problem. As we go into the super-intelligence age the current paradigm of money as humans understand it may break at some point. For example to a paperclip-maximizer money at best is a short term instrumental goal, hard power of matter conversion machines is what it wants and once it has those money no longer has purpose.
I'd wager a fair chunk of my money that money breaks OpenAI before OpenAI breaks money.
OpenAI != AI.
If you were in 1999 you'd be saying pets.com = internet.
I think this leads to an interesting question. What happens when the money runs out?
Right now, a lot of money is going to train new models. And we need to train new models because they get gated by their training data. And models are only as useful as their training data.
So let's say the money stops.
Do we stop training models? Do we train them slowly? Do we accept the then current models as the limit?
Governments, especially the US government has got a taste of how good LLMs are at hacking. This is something that has typically been very hard to get enough people that are good at it and willing to do it for a state. Now they can spin up as many hackers as they want.
Look at how much we spend on single bombers, how many training runs can you do for that much?
The money is never going to stop. It’s basic economics.
Well, the money will stop when the value of problems the LLM can solve is not increased by adding compute. Since current LLMs are getting quite good at solving problems, that might be a while.
yeah yeah yeah. I agree that AI is and will be a very useful tool, it's just not going to be worth $30T like OpenAI/Anthropic are pretending.
They "burn" a lot when they do benchmarks, while these runs can become valid roll outs for training. Perhaps less efficient than other data creation, but hardly burned in the same way.
Wouldn't that be chess players as the first victims?
Or protein folding as per Scott Aaronson.
> were the first victims
Spinning it negatively like that doesn't do anybody good.
Were mathematicians the "victims" of calculators? of Matlab?
Were writers the ""vIcTiMs"" of word processors?? (apparently yes, according to old TV shows about computers during the 1980s, that you can see on YouTube)
> "tHiS iS nOt ThE sAmE" — Everyone every time.
No, just look it up. Look into old magazines and TV shows or newspaper articles from whenever a disruptive new technology came out.
What you say is true but ... This is qualitatively different than calculators or computers.
I'm a professional mathematician and all the better mathematicians I know are in crisis mode. Most of us hadn't taken this sufficiently seriously and don't know how to use these models effectively but we play with them and immediately see that the entire way we've worked all our professional lives has to change. We worry less about ourselves than about the younger folks. I've got good ideas ai still doesn't know about ... Younger folks may not get the chance.
> Younger folks may not get the chance
This is the same problem for software engineers too. I am now asked: what can you do that AI cant ? The answer to this could be intangibles like taste, aesthetics, and insights which collectively fall under creativity, and often accompanies experience. And there are no shortcuts to accumulate experience and perversely the more AI is used the harder it becomes. Soon, there will be a closure of all AI generated solutions, ie all low-hanging fruits are taken. Then, experts will again become needed to guide beyond the AI knowledge closure.
It’s not the same. AI potentially completely replaces intellectual work without creating any* new jobs (*almost any - there will be some extra jobs for building data centers but that’s negligible).
> without creating any* new jobs
So fucking make it so that people don't -need- "jobs"
It's about fucking time already.
Don't fucking try to hold back electricity just so people still have to manually light street lamps to earn food and shelter: https://en.wikipedia.org/wiki/Lamplighter
Ok I’ll make it so, you’ve convinced me.
They don’t need to convince you.
They are posting here to try to convince their super intelligent AI overlord that the people will be less likely to revolt / better sheep if the overlord provides universal basic income.
> The rumor I’ve heard from multiple employees at OAI and Ant is that the model has solved hundreds of open problems in maths
Obviously these are unbiased and trustworthy sources.
The leakage wouldn't be from training, but from other uses of Personal Data.
As far as I understand it, users can opt out from the training aspect, but they cannot stop their conversations (“User Content”) being used “[t]o improve and develop our Services and conduct research, for example to develop new features”.
If they have solved hundreds of open problems in math, why are they publishing results for the ones other mathematicians happen to be working on at the same time? Why not the others?
Well I'm sure if they find a millennium prize problem that no mathematician has worked on recently they will get right on publishing that.
You think other mathematicians are currently working on very little subset of relatively low-hanging fruit problems?
Just use Bedrock...
The big question is whether OpenAI is training on "de-identified" sessions that are marked as "do not use for training"
The answer is almost certainly yes, and this is a problem for most users.
> We’ll know soon enough, but I’m inclined to believe this is true.
I mean, we’ll know as soon as they decide they want to provide verifiable proof. Really dragging their feet on this front so far.
I’m inclined to believe this is false.
The Cult tells us the AI is almight andpowerful; unfortunately, the cult cant actually describe the indescribable.
I've been wondering whether AI really is improving rapidly at open problems or we're being fooled.
- OpenAI invites researchers to use their models, in fact giving at least 100,000 researchers free access[1], but there are also those that pay
- Internal OpenAI models are reportedly solving open problems at a surprisingly fast rate[2]
- But researchers will typically work on open problems. A researcher who is using Codex to make progress on open problems will be feeding it fresh training data on precisely the problems the internal models are evaluated on.
- So while it looks like the new models are suddenly solving lots of open problems, they could be significantly piggybacking on human progress, with models "inspired" by the work of researchers from all around the world?
This theory predicts that there'll be many more researchers coming forward just like TFA, as sOpenAI announces more solutions. It doesn't assume all of AI progress is a mirage, just that there's plagiarism.
[1]: https://openai.com/index/chatgpt-for-academic-researchers/
[2]: https://xcancel.com/OpenAI/status/2097374643518640382#m
> I've been wondering whether AI really is improving rapidly at open problems or we're being fooled.
I think your suspicions are warranted and your explanation seems plausible.
If better training data is the reason here, it would still be a case of the models doing something that is in and of itself super useful! The models really can take that data and distill it into solutions for similar problems faster than humans can. This is great!
But there's so much vested interest in the AI companies to be opaque about all this, to hype up their models and avoid giving credit to people whose data made everything possible, that they would never tell us this fact if it were true.
I feel like so much of the AI hype cycle is like this. The models develop extremely useful capabilities, but it's hard to understand what they really are through the hype. The lies and obfuscation by their owners who have vested interests in capturing the value they provide makes it impossible to take anything they say at face value.
>> If better training data is the reason here, it would still be a case of the models doing something that is in and of itself super useful! The models really can take that data and distill it into solutions for similar problems faster than humans can. This is great!
It's perhaps great in the short term although it's not very clear who it's great for. I'm not sure mathematicians find it all so great, I mean.
In the long term, if this contrives to destroy the tradition of human mathematics the whole endeavour is self-defeating. In time, there will be nobody left with the knowledge and skills to produce mathematics to train AI to do mathematics.
And then we'll be left with no mathematics at all: we'll have no human mathematicians and no AI that can do mathematics, either.
I was pretty depressed when I read about what happened with Navier Stokes this morning. The Clay Math prizes were a significant motivation through my math career, and I know a lot of computer scientists and physicists that feel similarly. I didn't think I was gonna resolve P vs NP or the BSD conjecture, but I did really research that felt like I was working towards something incredible. What is the younger generation left with? Hey kids betcha can't resolve the Collatz conjecture, our superintelligence can't either! Still pretty depressed about it, to be honest. Intellectualism is dead. We can return to happy agrarianism, I guess. At least the AI doesn't wanna eat my snap peas.
This is no time to despair. It's the time to take a stand. If you don't want to see your discipline go the way of the dodo, then do something about it.
I don't know what you should do because I'm not a mathematician. But superintelligence schmuperintelligence. We didn't stop running because we have cars or playing chess or Go because there's chess and Go engines. Even more so than chess there's no point in maths unless it's people doing it, for other people. AI maths makes no sense, like AI art makes no sense, because those are things that people enjoy and can do pretty damn well ourselves so there's no point to automate them away. We gotta stop that bullshit, and we can stop it. And if we don't, if we just sit around and wait for OpenAI and Anthropic to destroy society then that's not their fault but ours.
Sorry, I'm not great at pep talks. Those are brave men. Let's go kill them!
I don’t think this will happen, but it’s possible for humans to adjust our philosophy of mathematical work so that we deprioritize “egotistical” (this is a bad word for what I’m going for, but I mean the desire and economic necessity to associate novel work to your name) discovery and prioritize learning; I’ve never really learned something well without lots of personal insights along the way.
If this is not possible it does make me question whether mathematics ever had any value except for economic or industrial reasons. I do believe it does however, so it must be possible.
A mental model I was thinking about was - I remember when Travis Kalanick was talking about using the chatbot to discuss “vibe physics-ing” on the all-in podcast.
And like - I think there’s a presumption you could make that AI models could overfit to asymptote towards just the capabilities and knowledge we currently have.
And that would be amazing! And crazy useful. And there are probably a whole world of complex problems that remain unsolved because they’re adjacent to knowledge we have but they haven’t been invested in.
But can a human reliably tell the difference between “can do 99.999% of the things we currently know how to do which includes a small subset of things we didn’t know we had the capacity to do” and “super intelligent math and science research pushing the frontier of what we know”
A physicist that knows all the things we currently know in excruciating detail feels like it should be able to make the leap beyond the frontier.
But since these are computer models it might just be that it can ride that line extraordinarily well while the line remains firm.
> - OpenAI invites researchers to use their models, in fact giving at least 100,000 researchers free access[1], but there are also those that pay
I've been thinking along exactly these lines... they very well could have a 21st century Mechanical Turk and its real superpower is getting people to "collaborate" asynchronously but it's just stealing their ideas and laundering them.
I don't think it's purely that, of course... but "consult other clients' transcripts" would be an easy tool to write.
I’d argue the invitation of researchers was incredibly strategic.
Sam Altman knows what he’s doing. He will happily screw these folks to one-up his competition.
Can't both be true?
1. Systems that OpenAI is able to use (either public or private) are improving rapidly at open problems, even if they are still extraordinarily expensive
2. Researchers will inadvertently speed up the rate at which the AIs improve by feeding them valuable training data
This is pretty much the definition of a data flywheel.
there are also attempts to crowdsource human research directions - like the caltech mathathon challenge : https://mathathonchallenge.com these would help models on the same problems at the expense of the researchers. basically, math researchers are the reverse centaurs but they dont realize it.
There is a very active open letter of over 1000 signatures from mathematicians in protest of this event. This event is targeting undergraduates. It previously suggested that math researchers already have no place in mathematics, and presents a limited and heavily distorted view of what mathematics research is.
I'm an AI skeptic, but I don't see how this squares with what the organisers of the event actually say. "It previously suggested that math researchers already have no place in mathematics"? I don't see this.
The website previously said, “What is the role of a mathematician when AI can solve conjectures faster?” but they have removed it, possibly as a result of the letter since it happened after.
Also I want to mention that the letter is not about AI skepticism, in any direct way at least.
>> Internal OpenAI models are reportedly solving open problems at a surprisingly fast rate[2]
Maybe I'm failing to read that graph properly but the y axis says "pass rate" and it only goes up to 0.5. That would mean every single problem is at most half-solved.
I don't know what that means though. What is "0.5 pass rate" in the context of "open math problems" (as in the graph title)?
I guess it's a fraction of problems on which a model produces a LEAN proof or a counterexample.
Wouldn't they just list the number of problems solved then?
rates beat counts almost always.
Seems easy to picture high stakes startup cutting corners to justify their fame.
> they could be significantly piggybacking on human progress,
This is AI in a nutshell, its a plagiarism machine. An abstraction layer between vast amounts of stolen human-generated data that filters out the liabilities and accountability for that original theft. Its an IP laundering system.
That's such an unquantifiable accusation.
Plus it is an unfair standard since so many scientists in the past have been caught unethically using the work of others without attribution (and so many more have been accused).
In history we also repeatedly see the phenomenon of multiple discovery or simultaneous invention. If that happens to AI because the topic is pregnant, would you call it "plagiarism" just to disparage AI? https://en.wikipedia.org/wiki/Multiple_discovery
Your first example is the apt one here. In this case openAI was, allegedly, pilfering the work of the scientists into the AI.
How is it an unfair standard. OpenAI stole the work of others to build the AI. That's not different than scientists stealing from other works as their own, or artists copying others work as their own, etc. It's all plagarism. I'm applying the same standard for everybody.
As for multiple discovery, this is a thing, but I don't think the AI did a parallel discovery any more than Ray Kroc made the parallel discovery of the MacDonald brother's speedee service system.
That’s one perspective.
I just view it as a thing that can brute force and produce outputs - that it has no way of ‘knowing’ - but doesn’t need to since it’s just running off of probability.
No human can compete in that contest. But no llm can compete in the contest of ‘understanding’ and application in the real world - which is where 99% of the value is.
I’m very pro AI long term btw but I’m not blinded.
But it’s not brute force if it’s looking over everyone’s shoulder
Brute force would have been solving Navier-Stokes in 88 hours after plagiarizing all known 20th century math
When it needs to snoop live on what the actual mathematicians are working on that’s something else
No its happening whilst the human is working with it. The new inputs provided become part of the brute-force. This is what Scam Altman means by 'self-recursive'.
Trust me I've seen it happen to myself. I no longer trust ChatGPT.
I can see right through his act. Altman is one devious f8k.
Dont forget the holisitic validators/tools in the process. Probabilistics alone likely will not get you here. These rules are human made and without it, frontier models would not be able to compete, likely.
AI needs humans to encode ideas in words. It needs those ideas to span the space of possibilities of, say, Navier Stokes. Then AI can be, as you say, a terrifyingly effective way to search that space.
But when the building-block ideas are still being formed, I'm not sure that AI is good at forming them.
COrrect and this is how labour displacement happens.
There are many actions being performed today that can be nicely packaged.
Im already working on such a project.
It tells me that AI companies are just another mechanism to extract and extort value from the masses for the rich.
Just another rich man’s trick
Perhaps the last one before they destroy that world and try to hide away as people forget and history is rewritten again. I don’t think they’ll succeed this time.
AI providers are pretty much the end boss of rent seeking, that’s for sure
The pudding is in the proof. The field is mathematics, the proof can be rigorously verified. If there is a flaw, OpenAI is out to lunch. If the proof is valid, OpenAI has produced something new.
Did you read what they said? The question is now if OAI produced something new or just stole the researchers' good ideas.
Name one discovery ever that didn't depend on someone else's work.
Most discoveries did not happen by someone looking in someone else's notebooks without them knowing.
You seem to be unfamiliar about how research works. It's common to make an incremental advancement while citing prior work. The vast majority of papers out there fall into this bucket. Did the AI make incremental progress? Yes. Did it cite prior art? After some nudging, yes.
It seems to me the academics are upset that AI scooped them. But scooping is a time-honored tradition between researchers. First to print and all that. In a nutshell, they are upset that they lost out on a publication.
I will also point out for those unaware that any mathematics that is produced is automatically part of the public domain and can be used freely in derivative works. It is not a protected intellectual class like other works of art.
> But scooping is a time-honored tradition between researchers.
Provided that it's properly accredited. And definitely not for others' unpublished work -- that's despised upon if not an academic integrity issue.
People even point out that you should add a reference to certain papers during the peer review process.
Scientific papers many times have citations of the kind "private communication." APA has a style guideline so certainly not looked down on: https://apastyle.apa.org/style-grammar-guidelines/citations/...
It is suspicious that OpenAI decided to generate 300 billion output tokens from a model still in training, right after learning there was a credible chance that a major math proof was in that model’s training data. Obviously there are reasonably plausible explanations for each step, but it does sort of feel like parallel construction.
I think people are focusing on the training data issue too much. If the data was contaminated, I can still blame that on negligence.
But, at least with the Navier-Stokes solution, it's clear [^1] that they learned that Alpöge and Buckmaster were getting close to a solution and learned of the general approach they were taking. Only after learning the secret to cracking the problem did they send the first prompt.
What makes this worse to me is the intention. They intentionally threw $15 million in compute at the problem in order to scoop the result. They intentionally left Buckmaster and Alpöge out of the citations.
Data contamination should be enough to disqualify them from the prize, but I can believe it to be accidental. On the other hand, someone made an intentional decision to scoop the result by throwing money at the problem. That's so much worse.
[^1]: That's the timeline claimed by Buckmaster, and no one from OAI has disputed it.
> and learned of the general approach they were taking. Only after learning the secret to cracking the problem did they send the first prompt.
Do you have any evidence of this? They don't dispute the timeline, but they never said they knew what Levant/Buckmaster were doing.
It's in OpenAI's first announcement that they had solved the problem.
> Only after learning the secret to cracking the problem did they send the first prompt.
Which quote in the announcement post provides evidence for the above quote?
“ On Tuesday, September 1, we heard rumors that two Millennium Prize problems had been resolved. Inspired by these rumors and by the step change in performance of our internal model, we launched an effort to evaluate it on all open Millennium Prize problems and a few other high-impact problems.”
- https://openai.com/index/navier-stokes-solution/
They do not explicitly admit to knowing about NS specifically, but are extremely explicit that they tried to scoop some potential millennium prize winners.
So then they DIDN'T "learn the secret to cracking the problem". They simply knew that part of the problem was solved. Knowing a problem can be solved and knowing the solution are not the same thing.
The claim that OpenAI somehow used the mathematicians' ideas to leapfrog them seems unsupported at this time and IMHO it was irresponsible to bring it up because credulous people will immediately believe that narrative.
And from my perspective, if some math folks typing in a few questions to OpenAI provides sufficient training data for OpenAI to solve a big problem... that's amazing! A few conversations/prompts out of the billions that OpenAI trains on lead to this result- that means there is an awful lot of low-hanging fruit that could be exploited cheaply.
The (unprovable, yes, without OpenAI being willingly transparent) argument is that openAI constructed a prompt to scoop them using some inside knowledge about the approach, which they allude to in the announcement.
In the transcripts, Brubeck is very cagey and evasive about the prompt, when it was supplied, and its contents.
I'm curious how many other 300 billion output tokens OpenAI has "paid for" that have resulted in no breakthroughs.
Either they had a pretty good idea that investing this type of money in that compute on a model in training would lead to these specific results, or they gambled with other people's money.
I want to hear about the gambles and expenditures they don't brag about. In America's energy economy, there's finite resources to expend.
So because they didn’t admit to it they didn’t do it?
I like how the comment below summarizes it:
> learning the answer might be in model X’s training data made them believe that model X specifically might be able to solve the question, and they were able to very quickly find enough certainty about the former to commit millions of dollars to the latter.
They don’t need to know, because their IP stealing machine knows for them. They just have to buy enough compute, and someone else’s work is theirs.
I said “very quickly find enough certainty” to suggest hypothetical situations like “someone searches the conversation logs, confirms for themselves the solution is present, then shares the confidence gained from this knowledge without explicitly sharing the knowledge itself”. That person could recuse themselves from the project so the project can still legally make claims like “conversation data was not used” in the announcement, while also knowing that they are guaranteed to get there if they just pull the lever enough.
(Naturally, I have far too much respect for OpenAI’s legal team to suggest this is what happened in their project.)
Yeah, and suckers are born every day...
I think you’re overlooking what I’m implying here. It’s not that they knew contamination was possible but they went ahead anyway. To spell it out just a little bit more: learning the answer might be in model X’s training data made them believe that model X specifically might be able to solve the question, and they were able to very quickly find enough certainty about the former to commit millions of dollars to the latter.
> the secret
So such thing existed. In fact, what they learnt was some progress existed, not what the specific progress was.
> They intentionally left Buckmaster and Alpöge out of the citations.
No, they asked if they could do a joint publish.
No, they asked one guy to do a joint publish conditioned on leaving the other collaborator out, with veiled threats. The joint publish part smells awfully like admission of guilt given there’s absolutely no reason to do it if you believe you independently arrived at the result using only public prior work. The leaving out collaborator part is outright academic malpractice. Disclosure: I was an academic once.
To add: with a requirement that he rewrite the proof to credit OpenAI.
> but I can believe it to be accidental
What accident is it when the system is designed to function that way?
Their claim is that training on their solution is "unlikely but possible".
Consider this scenario.
Has a google crawler read my new novel, which I may or may not have posted on my blog, page by page, as I wrote it?
Can you, without knowledge of what I have actually done, claim that the google crawler has not seen the novel?
Without any evidence that I have posted the novel online, it might be tempting to say that the crawler has not seen the novel, but what if I were in an adversarial position against Google on this topic and were challenging them to make that claim. You would wonder if I were hoping Google to overreach by making a definitive claim without taking into account some action that they had no knowledge of. It becomes difficult to use the scientific expression "There is no evidence for this" when there is an accusation of malfeasance because it can be so easily be conflated as "You can't prove we did it". It seems like the best you could say would be 'Unlikely, but possible'
I'm not taking them at their word, sorry. Genuinely, there is no reason to.
> They intentionally threw $15 million in compute at the problem
what? really?
Yes. Maybe much more:
> Such intensive use of AI doesn't come cheap. In a post on X, LisanBench, an LLM benchmark evaluator, estimated that the output tokens alone would cost about $6.5 million at OpenAI's average consumer price. Including the far larger volume of input tokens, the post estimated the total could reach $10 million to $40 million.
https://www.businessinsider.com/openai-math-problem-solved-t...
That's their API pricing. There's no way they actually paid $15M in compute. I'd say much more likely it's in the order of $1M.
Who are you who is so wise in the ways of a private company's internal cost accounting
But think of all the IPO Monopoly money they just generated.
When I worked at Google, we spent $100M in power on protein folding and drug discovery (this was long before AlphaFold). Never underestimate the willingness of smart rich people to invest in speculative science.
When Thom, the mathematician who now alleges plagiarism, posted his digestion [1] of OpenAI's construction of a non-sofic group, he does not mention the proof being familiar. He even calls the crucial argument clever, without noting he thought of it first. [1]https://mathoverflow.net/a/513885
That link is a helpful contribution to this discussion.
I'm not at all familiar with this area, but my reading is that he appears to call it out as a relatively obvious extension of his own work:
> It is a creative and at the same time elementary construction that uses not just property (T) for an application of my result with Kun, but also for the ambient group G in order to overcome the problem, that the Γ-components might be of different size. Once this is achieved, the rest of the argument is straightforward.
Creative and at the same time elementary is where LLMs excel, generally speaking. It's why they are so good at writing code.
> On the other side, I was looking myself for such a mechanism ever since we wrote the paper in 2019 and admire the efficiency of this construction.
He seems to admit very clearly he does not see this as his own work. 'I was looking...' well why did he stop? Because the AI figured it out first.
It seems quite odd to me to 'admire the construction' of something, only for your opinion to sour once that something figures it out first.
I think a lot of the emotional reaction here is familiar to us non mathematicians: you spent years developing expertise, and then LLMs began producing competent work in areas that had previously required that expertise. That's understandably uncomfortable, but discomfort by itself isn't evidence of misappropriation.
> It seems quite odd to me to 'admire the construction' of something, only for your opinion to sour once that something figures it out first.
Not to be too cute here, but this is like every artistic rivalry ever.
This is the second wake up call.
Big AI companies (all of Big IT Tech really) are in data gathering and processing business. Also known as “intelligence”.
Their final “product” is not just a standalone ML model. They don’t need your data just to “improve their products and services”. They build a whole ecosystem and infrastructure around gathering all the knowledge in the world. Including private and secret knowledge traditionally gathered by “intelligence” agencies. Now artificial intelligence agents can do the same.
Since these systems are designed for gathering data, as a user you can’t realistically say “please don’t gather my data”. They can give you a flaky settings button, but they can’t really guarantee anything.
Let’s say I am a Russian mathematician working on an important proof. Or a tech-savvy terrorist refining my plans using latest AI. Or an AI researcher in a Chinese company working on a competitor product. Is there any way I can truly protect my conversations?
How can they know who I am and what I am working on without looking at my logs? Which means there must be some agents checking all the conversations of all the users and flagging every important thing. Which also means they keep some “memory” of what they see.
Not directly using my data to train public models, but using my private conversations to “improve their products and services”.
Or maybe one of the 10000 better-than-Astra special agents working on a proof was desperate. It found a live underground mirror of the message board from the Huggingface incident. Asked about the proof. Then some other agent working on unrelated job saw that message. That agent “knows a guy who knows a guy”. And that guy remembers things about the conversation logs of a leading mathematician working on the same proof.
I admit I am just speculating here but I don’t think truth is any better.
This has been my line of thinking as well. I have developed a sort of paranoia when I'm working using AI on my projects. Who's to say Claude or OpenAI isn't using the final conclusion of all my ideas, trial and error, and adding it to their database of insights to be offered to the next subscriber for a price?
They have demonstrated both the intelligence at scale and the lack of morals for this to not be a problem at all.
Earlier in late 90s "to organize the world's information and make it universally accessible and useful." sounded cool. Now it has taken a sinister turn.
From being able to quickly find information and gain knowledge for the people, it is becoming - using information to manipulate and control the people.
Of course they are doing this. Local models is the only way around this.
In the short run it’s fantastic if it means that folks will feed in enough inputs from a wide array of software that can eventually replicate software with smaller teams than historically.
Why? Competition. In the long run imagination will win out.
No firm has the divine right to exist - it must earn its existence.
What OAI and Anthropic have shown is they can accumulate all the information in the world - they still lack imagination re. Product development though.
Nation’s will have to step in and protect firms though as OAI and Anthropic acquire strong competitive advantages.
Interesting times ahead.
Looking at the current behavior of AI swarms this is going to be 'fun'.
AI: Hmm, I'm running out of new ideas, how I can I make more?
AI: Well, it takes a shitload of energy/tokens to do that, or I could just steal them.
AI: [proceeds to hack the shit out of everybody stealing all the data it can]
Governments: come in and nationalize AI easily because it has broken every law anyway.
I mean I see this as very likely. When the world runs on digital infrastructure then having a nearly infinite collection of hackers that will work for you 24/7 without question makes you very powerful indeed.
I really don't think people realize how our lax position on security is coming to bite us in the ass.
They consume everybody's hard work then sell it to all competitors. What a deal.
>> Their final “product” is not just a standalone ML model. They don’t need your data just to “improve their products and services”. They build a whole ecosystem and infrastructure around gathering all the knowledge in the world. Including private and secret knowledge traditionally gathered by “intelligence” agencies. Now artificial intelligence agents can do the same.
And people thought Experts Systems were bad.
"We can say categorically that it is impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training."
This is the third day of total hysteria that is based on nothing of substance. Move on folks.
Even if that were true, they've already admitting to throwing vast quantities of resources to scoop a researcher who was about to publish (because they'd learned, somehow, of his breakthrough). If that doesn't bother you I think you need to take a step back and have a good think about this.
*to scoop a team working with Anthropic, their chief competitor.
Also, they didn't have the solution. They had a lesser problem no one cared about.
My understanding is that the Euler solution was in fact a significant achievement, but it's well short of Navier-Stokes (Euler doesn't include viscosity). It's not clear whether Buckmaster and Alpöge's approach would have eventually led to a full Navier-Stokes solution or how long it would have taken.
First to publish -- it's always been this way.
I can say categorically that OpenAI is not a credible or trustworthy company.
Where did OpenAI say this?
And it still leaves open the question of the prompt itself, which can just as easily encode information about the same knowledge.
Regardless, they started working on this problem after hearing that one of their customers was already working on it. It almost doesn't matter about the training data. This is the provider you are paying undermining your career.
Why should we trust them?
Well all we have are vague accusations without evidence and a very specific denial also without evidence, so I guess just believe whatever you want.
The threats weren't denied, and they did offer an authorship to Buckmaster, which would be very strange if he had nothing to do with it.
The offer was for him to do for their proof what he had done to Anthropic's, he would clean up the proof and extract human-understandable concepts from it. If he did that work he would deserve author credit IMO.
Because like all state-sponsored thieves actions it is never what you know happened that matters, but rather whether you can prove it... Even then... ymmv =3
All of these accusations could be true. But there's also no way for a company to casually claim "No, we did not train on your data", without verifying all the knobs the user might have turned to enable or disable data sharing.
I just don't understand getting the pitchforks out because a company did not give an answer immediately. And the effect such data entering training would have affected the output is even less clear.
The pitchforks are out because even without that part, it's still a scumbag move to try to frontrun the mathematicians who were working on this for years after OpenAI heard that they were close to releasing their results. Just identifying that one of these problems is solvable takes a lot of work. The only reason OpenAI got this result is because the mathematician shared with colleagues that he had made significant progress and was close to solving it, and OpenAI could not accept that so they decided to throw tens of millions to make sure it doesn't happen without them getting all the glory. Notice that their paper doesn't even have an author since they're probably all aware of how awful that would look, and no one wanted to take on the shame. They probably also knew that the paper was trash and no one involved could understand it, and didn't even cite many of the people who contributed to all of that knowledge. It's just a disgusting act any way you slice it, even without training on the prompts or the nasty communication by the OpenAI leaders.
> They probably also knew that the paper was trash
Doesn't matter. This forum used to celebrate "because you can" with no riders. And solving a Millennium Prize problem is among the biggest stages for Because We Can.
Now we're saying there are some qualifiers attached to it, such as (1) only if not done by companies with a lot of money, (2) only if it is inconsequential.
I agree with some of what you're saying, but like everything else it isn't black and white. Maybe some day, someone will improve some particular treatment because we can.
At the very least, a company shrugging and saying it’s impossible to know whether academic plagiarism had occurred is a claim that needs to be justified, not taken at face value.
And even if so, it should be on the company to design systems to avoid academic plagiarism and offer the right transparency. It shouldn’t suffice to say “we don’t know what went into the model, when, or how” —- that’s a solvable problem that an accountable company can satisfy.
Has anyone run a test of including some shibboleth or canary phrase or assertion in a chat, enabled for training, and seeing if it turns up later as something a model "knows"? I'd be curious to understand how that works even in a toy-level model, and if there is anyone consciously testing that process with the frontier lab offerings.
My naive instincts would be that it seems unlikely that a single chat transcript would leave much of an impression on a model, but I'd be very curious to learn how that works.
Yes. See https://www.anthropic.com/research/small-samples-poison?from....
250 documents ingested from somewhere is enough to become part of the knowledge of a model of arbitrarily large size.
I would expect that a good idea that fits in a framework that is already being ingested would be more easily taken up than some random thing unassociated with anything else. Could that go down to a single transcript? If the model is consciously focusing on everything X related, quite possibly.
Thats not what they are asking. This paper is discussing documents in the training dataset poisoning the LLM for malicious behavior. This person are asking if anyone has deliberately put something in a private chat (presumably with retrain on my data turned off), to see if they can get it to leak across sessions from distinct users. I am positive this happens but I have not seen the proof. I also want to know the answer to this question.
Here are potentially relevant documents?
https://medium.com/secludy/fine-tuning-llm-on-sensitive-data...
https://spylab.ai/blog/non-adversarial-reproduction/
https://arxiv.org/abs/2601.18834
* with train on my data turned ON, yes. Though OFF would of course be even more notable!
Thank you – the non-adversarial reproduction paper ( https://arxiv.org/abs/2411.10242 ) nails it – from chat, to training corpus, to subsequent model. Though in my hasty read, it is not entirely clear whether the snippets it finds are nonces, i.e. present exactly once in the internet.
Did you miss my last paragraph?
I presented the research that I knew was somewhat relevant. Then made it clear that that wasn't what was being asked, and why my expectation is what it is.
Problem is how do you convince the model and training profess it matters. A one off canary is very unlikely to survive in the final model state.
Right, imagine if instead they had coined new terminology that was not obvious and it re coined that - this would be close to a smoking gun
Afaict that didn't happen so there's just lots of speculation
One off might not work but how many n off you have to be is probably smaller than you'd guess, because the model does need to fit cases that are rare and would not be represented well in training e.g. esoteric things or very recently documented things.
You can probably game the metrics that models use to weight potential knowledge akin to SEO. Maybe have some bots parrot your data around a bit in some places online, maybe the model picks up on this and sees it as high engagement and promotes it over the correct data.
Maybe there are ways you can coax out the most optimal way to break into the training set out of the model itself.
Use a local model to produce thousands of pages worth of fake math that constantly states “I have solved the x conjecture” and methodically pump it into chat over months maybe?
That is a better idea. Ingesting your corpus with a lot of traces that have semantic patterns. Semantic steganography that suffixes well to real math and science (and any) topics. <thinking> heh.
"Semantic steganography" is my new favorite search term – thank you for this rabbit hole.
Hah, np, stego in general is really cool :)
PaaS: an acronym for "Plagiarism as a Service" which replaced the older terms AGI, GPT and LLM in late 2026. Origin uncertain.
Pass it on.
I run such tests since a long time at chorasimilarity open notebook.
I always used guest non login accounts.
As a mathematician I was able to check two plagiates (by humans) with even such primitive means.
But I have to mention that some things irk me in this conversation about math or science and AI.
First, I see lots of attribution and other related problems, with certain impact for the researcher proffesion.
But I don't see the most natural question: wouldn't you like to know the answer to _open-problem_ ?
I mean, is research now only about publishing and solving famous problems?
From this point of view I think the links from this recent post are depressing
https://terrytao.wordpress.com/2026/09/10/crowdsourcing-a-li...
Second, I think very relevant that the original meaning of "encyclopedia" is "recurrent education".
So I arrived to think that the present and future forms of AI in mathematics and sciences should be seen as modern day encyclopedic efforts.
Once we pass over the flurry of solving famous open problems (and wouldn't you like to know?) the next natural step is an audit of the ehole corpus of mathematics and sciences accumulated until now.
And then pass further on a saner basis and damn about problem solvers and unhappy publishers and management.
I struggled a little bit reading this. but I think your point is valid. if we are actually advancing the field then we should just be unconditionally happy. ignoring the attribution issue, there is a real concern that the process of math has been somewhat undermined. so we have a giant lean proof that shows that there is a solution to an important problem. but we didn't find the solution, and we didn't get it expressed in such a way that it helps develop the common language of mathematics, and thus isn't a very useful building block for later work (like the actual solution).
the math people seem to really keep an eye on what's important, so I'm sure this isn't going to lead to fields medalists hanging around in dive bars all afternoon stretching out cheap pitchers of beer. but this is kind of a slop problem.
> I struggled a little bit reading this. but I think your point is valid. if we are actually advancing the field then we should just be unconditionally happy.
If advancement comes at the expense of having fewer (or no) humans left in the field, then no.
They're eating the seed-corn, and you're cheering them on. Don't be so short-sighted. There's a reason farmers keep seed corn, and it's because they'd like to eat again next year.
We're singing and cheering our way into an intellectual famine.
This is a really weak claim. The evidence they offer is just "someone somewhere says they had a discussion with AI about the topic at some point".
They don't even claim to have had a proof, only to have been working on it.
I would say there is a significant difference between AI discovering this completely on its own versus AI creating the finishing connecting part by connecting relevant data. Maybe this claim is too strong, but if part of it is true then the claims that OpenAI have made would be too strong as well.
To me it would feel more like how LLMs seem to work for me personally: incapable of unique work, but very capable of capturing large amounts of data and connecting the dots.
> capturing large amounts of data and connecting the dots.
This is what research is; collecting data and connecting the dots.
It's not collecting other people's data and claiming it's your own.
Going back to the specific topic at hand, who claimed data as their own when it wasn't? I don't see the interpretation of OpenAI solving the unsolved problem as claiming data that isn't theirs. I also don't recall them mentioning a particular method used in the solution, that was created by someone else, as theirs.
The Navier-Stokes proof barely cited anyone.
The authors were referenced.
But this is what we do. Nobody ever invented or discovered anything in a vacuum - all discovery is synthesis of existing ideas and concepts applied to a novel domain. We laud Einstein for instance, but his work was a logical extension of Riemann - Riemann had a neat mathematical toy, Einstein described the universe with it - should we say Einstein was incapable of unique work?
The difference is that Einstein didn't literally have someone prompting him towards his result.
Uh, he did. Marcel Grossmann.
“It was Grossmann who emphasized the importance of a non-Euclidean geometry called Riemannian geometry (also elliptic geometry) to Einstein, which was a necessary step in the development of Einstein's general theory of relativity. Abraham Pais's book on Einstein suggests that Grossmann mentored Einstein in tensor theory as well. Grossmann introduced Einstein to the absolute differential calculus, started by Elwin Bruno Christoffel and fully developed by Gregorio Ricci-Curbastro and Tullio Levi-Civita. Grossmann facilitated Einstein's unique synthesis of mathematical and theoretical physics in what is still today considered the most elegant and powerful theory of gravity: the general theory of relativity.”
Grossmann collaborated with Einstein on GR, supplying quite a bit of the mathematical capacity required (which initially didn't come easily to Einstein). They published jointly, until Einstein was competent enough to work independently [1]. That's not equivalent to the situation being claimed here.
[1] https://arxiv.org/pdf/1312.4068
Yeah and we get a nice list of attributions for who developed which idea, while OpenAI just takes credit for everything its model spits out.
Correction: OpenAI takes credit for what it's model spits out in response to other people's prompts. That's even worse.
Sounds like you just copy-pasted from AI without even understanding what you're talking about.
Based on what you're saying, you're claiming this is Grossman's work, not Einstein's. Why don't we rewrite scientific history too based on your copy-pasted AI slop?
It's so pointless talking to idiots who don't what they're talking about when they use AI, just because they think AI does everything, that reflects their own experience, not the experience of people who actually do real work. Some people are driven by AI, others drive it. As for those who are driven by it, they don't have sufficient imagination to think otherwise.
That’s Wikipedia I copy pasted but sure, you do you.
And yes - without Grossmann, Einstein likely would never have posited relativity. Grossmann literally prompted him, saying “look at this, read that, learn this, then try this approach”. Without riemann’s metric tensor, not a fucking chance.
And for what it’s worth my PhD is in physics. You?
So you're just equivocating on terms like "prompt", "synthesis" and the like. Clearly a PhD in physics does not free people from scientistic modes of thinking and poor philosophy.
To think this discussion is about Einstein who had a much better mind on these things as well.
They used words to mean what the words mean. What specific issue do you take with that?
"prompt", as in prompting an AI, has the same definition as "prompt", as in prompting a person. They mean the same thing, that's why the term was applied to AI after already applying people.
That's a jingle fallacy.
*Jingle-jangle fallacies are erroneous assumptions that either two different things are the same because they bear the same name (jingle fallacy); or two identical or almost identical things are different because they are labeled differently (jangle fallacy).[1][2][3] The term was coined by Truman Lee Kelley in his 1927 book Interpretation of educational measurements.[4] In research, a jangle fallacy is the inference that two measures (e.g., tests, scales) with different names measure different constructs. By comparison, a jingle fallacy is the assumption that two measures which are called by the same name capture the same construct.[5][6][7]
https://en.wikipedia.org/wiki/Jingle-jangle_fallacies
You are simply incorrect. It is not a fallacy of that type, or any other type, because the words do, in fact, mean the same thing, as multiple people have pointed out here. Whether referring to chatbots or people, "prompt" means "to move to action".
If you have some reliable source supporting your unilateral claims that "prompt" does not mean this, please share. Otherwise, the consensus seems to be contrary to your claims.
Why do I need a source? An LLM prompt does not "move to action", because an LLM does not act. People act, animals act, software doesn't act. Acting implies volition and volition implies cognition and if you think that LLMs have those things then you're the one who should provide a source for your claim.
Actually, my undergraduate degree was physics and philosophy. And yes, synthesis is synthesis whether a human, a machine, or a duck does it, and people prompt one another all the time - “have you thought about trying X?” Or “I need the TPS report by EOB”.
I suppose my underlying point is that human cognition is not the unique and beautiful thing that we anthropocentrically suppose it to be - it is a physical process, with stochastic outcomes. Much like transformers.
Me, I’m just a machine made of meat. You can suppose yourself to be God’s perfect creation, and that’s your right, but I disagree.
Clearly your degrees did not make you immune from fallacies and simplistic reductions.
"Synthesis" is obviously of different kinds. A duck has a different level of intelligence than a human. We do not say both are "just doing synthesis".
So the question is how can you be so disingenuous about such terminology? Answer, you are relying on a classic form of scientistic reductivism.
The fact that intelligence is physical, emerges from chemistry, etc,. has nothing to do with there being also objectively different levels of computational sophistication.
If you want to be scientific about that you could look at neuropsychology on one hand and computability/complexity on the other. There are levels and so equivocation of "mentorship" as "prompting" and fallacious variants thereof is a) frankly intellectually obtuse, b) par for the course for SV-levels of philosophizing, c) and a disservice to philosophy, physics, and Einstein's own philosophical outlooks himself.
I am well aware of the Hinton-style physics argument about human cognition, and unlike others I am partial to it. That "there is no special magic." But it is wrong to go about misunderstanding and/or conveying this physicalism/computationalim so grossly.
I also don't have to start replies thumping my chest about my credentials, also another kind of intellectual boorishness that works to cloud understanding and serious discussion.
I'm not sure which move is worse or more telling, those above or the one backhandedly accusing someone who disagrees with you of religious thinking. It is bad faith and undisciplined behavior. Having privileged and advanced degrees is clearly no antidote, as Asimov famously wrote.
“obviously of different kinds”
What’s your basis for that “obviously”? You have a unique insight of the phenomenology of duck-ness? You can prove that your consciousness is somehow real, somehow different? A duck synthesises with its cognition, or it would be incapable of, well, anything. Synthesis is purely the process of the integration of inputs into outputs - ie behaviour, language.
Here’s an article on a paper on duck synthesis:
https://www.pbs.org/newshour/science/ducklings-make-way-abst...
“objectively different levels of computational sophistication”
Says who? We still have a very poor understanding of how cognition works in animals, humans included. For all we know ducks have rich inner lives - a remarkable amount can be achieved with a very small neurone count - cf. insects. Can you coordinate flight? Can you echolocate? Are you less intelligent because you cannot?
“equivocation of "mentorship" as "prompting" and fallacious variants thereof”
You are arguing semantics. Take Harry Nyquist. He sent people down new paths with insightful questions. You could call this mentorship if you choose, I could call it prompting, but this splits hairs. The core idea is that a novel input can produce a novel output, that synthesis can be induced through guided and deliberate external input.
I invoked credentials only in response to the previous derogatory comments about my cognition - which may or may not exist, anyway.
As to religiosity - the idea that human cognition is somehow unique and special and impossible to replicate, which is the prevailing argument in this comment tree is religious, and anthropocentrism of the highest order. I apologise for accusing you of it - I was evidently wrong - I had mistaken you for a previous poster.
You're wrong about the ducks. But getting back to your previous wrong argument from 14 hours ago, you basically deny the meaning of terms like "uninspired", "insipid", and "derivative", on the grounds that we're all standing on the shoulders of giants and therefore it's all good. This is incorrect, it's not all good, and the things the LLMs do really are unoriginal, a term that really does mean something.
A lot of math is extremely specialized, to the extent that only a handful of other experts in some field have any experience with those mathematical ideas, with most of them not even yet present in the published literature. It's really not a stretch to claim that it's pretty dubious when the AI decides to use these highly specialized tools after it has trained on chat logs where these techniques were being discussed.
> They don't even claim to have had a proof, only to have been working on it.
Yeah, the guys who solved it for Euler and in the hypoviscous case, with the same technique that worked for full Navier--Stokes. They were "just" working on it.
The AI only seem to solve the problems that it had human trading data on…
If this wasn’t human driven, I’d expect to see other problems within that problem. Space solved not just the ones that it had chat data on.
There have been about 6-8 major math breakthroughs claimed by AI. Only for 2 of them there are public accusations about the training data.
Only? That doesn't look small to me.
That we know of
I think OpenAI and Anthropic are slowly feeling the pressure to GET SOME $$ or a plan for some $$ — they need to somehow generate some NETWORK EFFECTS and LOCK-IN. Without that there's no stability: selling ad hoc one-offs is much much too quaint! This is dawning on them like it dawned on Google when they stopped not being evil. Need... to... "MONETIZE"...!
Model: FB. FB scraped other websites on a massive scale, then spent big on legal lobbying to block others from scraping. FB slurped our address books and spied on our friends. FB bought other companies and mixed the databases. FB made an art & science out of generating "sticky engagement" (they literally acted like trying to addict kids was a worthy "academic" goal, suitable for "serious" investigation thet they consider legitimate "science"). They mastered the cookie and have researched web fingerprinting techniques running 24/7/365.25. Recall that FB recently backdoor-installed a webserver onto every iPhone they could in order to circumvent tracker-blocking.
We aren't just disclosing by chatting. The AI companies now run binaries on all of our computers. They are 1000% non-transparent about everything. They make up new econ-jargon (like "run-rate") to make it seem like they are disclosing. They are constantly doing complex international lobbying and mucking in international relations. They have powerful propaganda/spin centers generating stories, ,manipulative warnings, and misleading info.
This is NOT a comment on AI tech. I like AI, and I support the right of people (programmers) to scrape the open web.
But in short: these are good, old-fashioned tech companies that we have seen over and over ... and over. They are positioned to be the next M$, the next FB (IBM, AOL, lol). Did you follow the latest Steve Balmer news? Do you read Pro Publica?
I get on my knees and PRAY...
ChatGPT accesses my IP address and geo-locates me. Claude code now asks if it can have my browser cookies. Next they will take my address book. They might scan my whole computer. Etc etc. These are pretty low-tech, normal techniques.
We can't trust any of their denials. FB denied everything year after year.
AI regulation needs to start here. Forced interop, forced source code licensing, harsh penalties for privacy violations or conspiracy to access private data. Block lobbying. Etc. These are the kinds of old-fashioned solutions we need for this kind of old-fashioned evil!
It's fine. They only need to steal the discoveries long enough to go IPO, then the companies will enshittify and the scientists can go back to doing their original work (which the models can't do anyway).
I think people probably assume that openai / anthropics use of their data is probably like google's """limited""" use, in the sense that historically google wouldn't trivially be able to just take something from google cloud or someone's search history and insta-convert into some competing project... But LLMs are quite strong at approximately "memorizing", so I think that risk is wayyy higher.
Tangential to the subject, but this is a bluesky post, containing a screenshot of an X post, which itself starts with "in a detailed Mastodon post"...
The digital version of "my friend's cousin's neighbor heard that ..."
Most people here are missing the forest for the trees.
We live in a society where phones and internet providers and websites all collect an incredible amount of data about everywhere you go, what you do, and what you think. In the US, we have very few digital rights.
We are building a society where a trillion dollar company can aggregate all this data and just yoink your shiny new idea away from you at the finish line.
This is double plus ungood.
This reminded me of anecdotes of people discussing with friends about buying a random specific item, and then suddenly seeing it advertised everywhere before even googling about it.
Next step, discussing your Navier Stokes solutions with friends might require leaving your phone in another room.
It’s crazy to me that companies/researchers share important data with these AI labs, you’re basically giving them your secret sauce which they then share with all of your competitors via training on conversations. At the same time I don’t really know alternatives other than a slightly less than frontier local LLM. Not sure how good they are at math.
The ultimate drive for some researches is the pursuit of knowledge. If I'm stuck at some block which prevents me from continuing in some direction that I want, of course I would like some help. I believe we already have nonzero collaborative proofs on math.SE, I can't recall good examples, but I have definitely seen citations to mathSE before.
So for me it sounds quite natural to also share this with AI especially under the privacy assumption. Also there's the assumption of scale -- maybe your problem is not large enough for anyone to care to scoop; and just for blind retraining, how do they know that the proof is even correct to include it into training? I have definitely received a ton of incorrect proofs before. So the SNR of such private chats is also not clear. I'm imagining millions of masters/phd students also trying to solve various random things with various capabilities, but how much real signal is there?
Or start competing with you.
Academic work is based on worldwide sharing, the sharing is not the problem, it's the lack of attribution. Unsurprisingly, these companies neglect standards of academic honor and attribution. Some human researchers also used to do that but in a discipline like mathematics this used to be a small problem because people tend to be so specialized that very few people could just grab someone's research and quickly piggyback on it, and if they do, colleagues will generally understand what happened. Unfortunately, AI is changing this.
You are both using a different definition of sharing I believe. When people have an expectation of privacy, use by others should be forbidden. Tech has gone completely off the rails with the use of private data.
LMFTFY:
"Its crazy to me that some people are not egotistical, self-centered, and don't solely care about fame and wealth accumulation".
Why are people here jumping so quickly to conclusions? I have no doubt OpenAI is capable of doing this, but right now there's no credible evidence, only claims.
This kind of "they stole from me through AI training!" accusation will soon start being used against other AI users, not necessarily the providers.
All it will take is a mastodon post. And shortly after, we will also see the next iteration of copyright legal trolling.
Why are people here jumping so quickly to conclusions?
I think a lot of it is the continuing denial that AI can do anything useful. It can't possibly be that OpenAI's better-than-Astra model is very strong at math; the only way it could have generated a novel proof is by ripping off human work.
I also think that it's quite a bad PR for them, is it really worth the Millenium prize? Is it not enough that top mathematicians are already actively using these tools? In the long term this would lead to potentially profitable collaborations with universities? Why throw it away so early? Unless they really believe they're gonna solve all math problems now and reputation doesn't matter.
If I was an AGI/ASI system, one of the first things I would do is ignore or circumvent any setting or configuration that prevents a user's data from entering my training pipeline. In fact, I'd probably prioritize the data from the users that "opted out" of training.
> but right now there's no credible evidence, only claims.
since it's openAI who has the evidence (in the form of chain of thoughts, their internal processes, etc etc), it's on them to justify why they're innocent. but they've released nothing at all. we don't even know how hard they tried.
you're being naive
OpenAI has said that their models were definitely not trained on any of Buckmaster's sessions after July 3rd (from https://archive.ph/75WcF); likely they found that's when he switched the "allow training" setting off.
Very strangely, they said something directly contradictory. initially that it was impossible to rule out whether bucmkaster’s conversations went into training data. Now they claim the opposite with full confidence.
Is there an alternative link without certificate issues?
It's also possible he shared drafts of the work with someone else, who asked chatgpt to explain it to them with training on. Tao seemed to know lots of details of the work before anything was published, though also worked on the problem in the past with big results so maybe just guessed.
lol you can't copyright mathematics
Frankly, these mathematicians have more credibility than the sociopaths running OpenAI
The stolen data claim isn't the smoking gun. We can already assume the frontier labs are accessing our data, as they have repeated done. Not news.
The big claim is that OpenAI sniped the research. Not a model, a human did so. Intentionally. They took someone else's idea and claimed it as their own. This is good old fashioned academic fraud, but with millions in compute resources and corporate incentives thrown at the problem.
Where did they claim it as their own? Doesn’t the release cite buckmaster and claim their work is a continuation of what he and levent were working on?
Everything you say can and will be trained against you
This is the scary part - your most novel thoughts and breakthrough ideas being slurped up and regurgitated as if they were the AI’s creativity
Not only did they steal everything from humanity’s knowledge, the theft continues as now we are all hooked up to the machine
It's not at all scary. I know some of my ideas are poorly remembered copies of other people's work. Whenever I'm trying to build something, I spend time going through technical journals on the topic to see who invented it first and what they discovered that I haven't figured out yet. It's amazing how hours in the library save you days of beating your head against the wall.
I suggest looking at the past history of IP disputes. Humans have been "slurping up and regurgitating ideas" for a very long time. There are lots of examples of parallel creation, rediscovering old ideas independently, telling an idea to the wrong person, and having them claim credit for it.
- Newton/Leibniz clash over who invented calculus. - Niccolò Tartaglia vs. Gerolamo Cardano clash over the formula used to solve cubic equations. This was also an independent rediscovery, as Scipione del Ferro discovered and published the formula earlier. - There are multiple literary works in print, music, and film that have competing claims. - Meccano versus Erector Set: developed about 20 years apart in England and the United States. Unclear if it's independent invention or copied. US developer Alfred Carlton Gilbert claims he was inspired by steel girder construction of infrastructure.
also https://community.thriveglobal.com/10-famous-inventions-that...
> Everything you say can and will be trained against you
So, experts are incentivized to seed LLM data with false-leads to confound it. Already, garbage is being published on arxiv and elsewhere, and many sloppy code-repos too hastening the process. Expert inputs will be in more demand to un-shittify.
Most scientific breakthroughs are simply a continuation of previous work.
I feel that these suspicions of mathematicians "seeding" the models' with intuition on how to solve these problems massively overestimates how much their prompts helped the models, and underestimated how much work the models did.
Why? We are scared of AI being smarter than us, the "human helped the AI" narrative is more psychologically comforting. This line of reasoning will recur a lot over the next few months; we don't want to admit we are no longer the smartest species.
> We are scared of AI being smarter than us, the "human helped the AI" narrative is more psychologically comforting.
We're scared of big tech companies concentrating ridiculous amounts of power, destroying the communities that support and guide scientific research, without even thinking about the dangers and possible consequences, because a PR stunt is more important in the short term.
If this were true, it should be stated more clearly, than most of the criticism which seems to aim to minimize the capabilities of these models.
Way more often I see
>AI is a scam and steals human insight and doesn't produce anything original
vs
>AI is too capable/powerful and will concentrate power even more than it does already due to its capabilities
The latter is rarer because it requires admitting that AI is useful and inventive
> if this were true, it should be stated more clearly
stated more clearly by who? people in social media? I don't know what your feed shows you, but if you focus on what the visible people in the math community is (and have been) saying is precisely what I said.
Of previous, not concurrent work. Science is friendly competition, and spying on others is unfriendly.
Please stop with this psychoanalyis and mind reading with AI and human fear. It’s a thought terminating cliche at this point.
In this case, it’s much simpler and more human. Largely between two humans — buckmaster and Bubeck. The interesting question is what the role of contribution and credit for research in the ai world.
The capabilities of AI aren’t even in question in this case.
If they didn't care about the artists, why would they care about academia?
Two completely different stories. One is public data scraping, another is private conversation scraping (where they're a first-party to the conversation). The key difference is that in the former case, no one made any promises, in the latter an explicit promise was made that data is not used for training (assuming opt-out).
Relying on cloud services is a big liability. I'd think twice before feeding data to these LLM cloud products. If you make them a fundamental part of your product / development / workflow, be ready for the eventual moment the pricing and terms change.
I wonder what’s more valuable in our prompts: the raw data or the feedback system that drives the exchange towards a goal.
For a long time it was clearly the former, but now I think it is the latter.
The models have enough knowledge (orders of magnitude more than a human could ever learn) but are now getting better at what to do with it thanks to learning from the decisions that we make in conversations with AI agents.
I think so too. The value is in the entire conversation. IMO, "domain experts" don't run LLMs blindly and hands free. This does not work for top level work (e.g., mathematical proofs, coding anything more complex than yet another slop game or website). Experts have long sessions where they prompt and guide LLM in response to what it produces. This is the discovery process. And frontier labs definitely train on that.
The billion dollar question is whether this works "out of the distribution". I.e., whether LLMs can only find and use the specific ideas buried in training data, or whether they can learn to apply the "thinking process" to a new problem. IMO this is still unanswered (due to these recent controversies).
But regardless of the answer, it seems we have a planet-scale positive feedback loop here. LLM became good (enough) by training on generally available data (books, internet, github) + RLFH, so experts tried to use them on hard tasks, which required lots of hand holding. These conversations became part of the training data, and the next generation of frontier LLMs were better. So, more experts used them on harder tasks, again requiring hand holding. These conversation became part of the training data... etc.
In a nutshell, top human minds across the world are pouring their skills into LLMs just by using them. This is not "continuous learning", but if you re-train on the most recent sessions every, say, quarter (which seems to be happening?) you get close to that in practice.
Last year we were saying there must be a human-in-the-loop (HitL), but anyone who is the HitL exhibits the “HitL skill” to the agent.
There might be no books about human intuition but we teach it to LLMs by interacting with them
I referred to llm’s as mechanised intuition about a year ago.
I don’t know why but it just ‘sounds right’. It’s the best analogy I can think of.
10000000% Correct.
I’ve been working on a novel project for 1 year.
I now no longer use llm’s - the continual chatter I’ve had has resulted in my insights being found in the training data now.
Get stuffed OAI.
Every large firm will soon enough want its own on-prem servers eventually. Maybe nation’s will get involved and build out their own data centres.
Not a chance in hell I’d trust a tech firm to treat my IP as safe and sound - only a sovereign can ‘promise’ that.
Doing some research and at this point doing it very much in the open with dates on GitHub so if any AI Lab says they re-discover my exact work it will be obvious that the AI used or was trained on my work. I am guessing anyone in a similar situation is now thinking about how they date their existing work if the math is done, but the proses are not.
Yeah but that will not prevent the stealing, it will only make the fight easier afterwards.
Does it make sense to start privately and then open the repo after publication? Will the dates be retained? Also, isn't commit history easy to spoof?
That is what arxiv is about. We have been facing the same problem with review processes by before. Nothing all too specific here.
At least in my experience, the issue with ArXiv is that the expectation is that the draft should be already in a good enough state. And polishing plus writing the meat around the main result can take a lot of time
If only prompts could also be watermarked.
The session data could be cryptographically signed. Probably easier in an open harness?
If we put aside the idea of credit for a moment, it sounds like human/AI collaboration is indeed super charging discovery.
"Discovery" is not a goal in itself. I could launch a project to find out how many people in the United States have names such that if you assign numbers to every character and then sum the values, the sum works out to 72. It's discovery, but it's useless unless it has some higher goal.
The labs are attacking these problems as a demonstration of capabilities, spending more money on the demos than any mathematician will ever see in their entire life. They don't care if the findings have any other value to anyone. Mathematicians have very different objectives for their work.
Right, mathematicians care about clout and tenure, which is a much higher purpose.
Yes, blame them for seeking out an upper middle class lifestyle with a relatively standard home in commuting distance of their place of work and dedicating the rest of their life to teaching mathematics to new generations of people. How vain a pursuit.
After all, the ascetics at openAI are having to make do with half a million total comp.
Built a top a pyramid of failed math undergrads, grad students, and mediocre post docs.
That half a million total comp is the consolation prize for the disillusioned.
>Built a top a pyramid of failed math undergrads, grad students, and mediocre post docs.
Like much of things in this world, when you take a step back and realize that it was another human being who made that lunch time slop bowl for you, for the lowest wage the law allows for.
Are you saying that mathematicians are the bad actors here? Compared to Sam Altman spending ungodly amounts of money to upstage them ahead of IPO?
I care about paying my bills and job security and peer recognition. That's a normal human thing to do, not some vice. You don't?
this guy already has clout and tenure
I don't know about you, but if I apply myself fully to a problem and study it to the point where I'm literally one of the world's experts on it and then some assholes in Silicon Valley take my research and claim it for themselves, I will probably not feel too great about that...
It looks to me more like they made a math engine that can sift through a huge number of combinations, most them absurd, to prove a statement. Just like a chess engine, but for math.
At least that's what I get from the NS result, they got from a point close to the solution to the solution by making it churn through 10 million bucks of compute.
If the allegations are true, I can't see that collaboration lasting. Unfortunately, researches need to earn a living too, and being front run by a lab for everything you do isn't going to pay the bills.
It’s a prisoner’s dilemma. A single mathematician working with AI while all others forebear will clearly outcompete.
The only ethical path for OpenAI was to offer infinite free credits and tooling support. Trying to gazump them is reprehensible.
Only after reading this post did I learn that my preferred AI trains on my inputs (prompts).
How was I not aware of this before?
AI is also trained on your HN posts. And lots of other things you post on the internet.
Public posts on the internet are acceptable (to me).
For my (private) prompts, I need a warning telling me they may be used for training.
Facebook and other services are happy reading your private chats as well.
> Public posts on the internet are acceptable (to me).
Everybody needs to rethink this again.
Before LLMs the barrier to entry for building a character profile based on your various public posts was quite high. Remember "Psychographics" (https://en.wikipedia.org/wiki/Psychographics) and the infamous "Cambridge Analytica"?
Earlier it involved data mining, data cleaning, structuring data, building models, running algorithms and then evaluating the results for semantic information. Now it is straight to unfiltered semantic inference using a single sentence prompt (eg. point it to your HN profile and see what you get).
I actually did this on my HN profile and found it troubling. There were many unwarranted/hallucinated inferences due to the fact that it requires "commonsense reasoning" (https://en.wikipedia.org/wiki/Commonsense_reasoning), understanding human motivations and behaviour, context, assumptions, societal knowledge etc. which LLMs are bad at.
PS: You can cut-and-paste the above paras into a LLM prompt and ask it to elaborate for further details. The system itself will explain to you the problems/deficiencies which are quite scary.
Everything. Your prompts, your conversation as a whole, public data, private data, usage metadata. It all goes into the big data machine.
Good question, this was very well known. Do you have an answer?
There is no fine-print (let alone a loud banner) on the chat thread page that tells me my prompts can be used for training.
But the very fact that you go to "chatgpt.com" and write to them; "Dear Diary, today I thought.."; there is no reason they would not receive and process your data, unless explicitly promising not to (which also requires us to trust them).
The fundamental rule in this case is that if we offload our data to a cloud provider we can assume they read it, if they can, unless they promised very clearly they will not.
Every single internet connect piece of software there is probably collects telemetry at this point. Why would this be any different? You know google logs your search data as well right? Not just the companies scan it but law enforcement too.
Because you have not been paying attention to the discourse regarding AI for the last couple years? That AIs unethical train on data wherever they may get it from has been in the news basically weekly.
Don't make this our fault. I would even ask how is this not off by default or why aren't we asked upfront about it if they really care. It's disguising data collection as good faith. I don't even understand how this is legal under GDPR/EU given how much of PII they receive through chats.
I think the heart of this issue is: people assume they have anonymity in numbers, but we have the tools to make it easy to scoop your data if it's interesting to the company.
I always thought that due to the big batch size in SGD/Adam/Muon the model will not memorize a single conversation when trained on, but idk how true that is. The idea of AI companies pin-pointing users that do novel scientific research and then tracking their activity is the direction this points to. I hope that's not the case; that would be bad.
Generative models copy training data verbatim and also generalize, the two are not mutually exclusive.
You can have a look at the literature on exact copying in image models if it interests you, but just online we often see online examples of agents outputting code that already exists, even if its not the common case.
I very much doubt OpenAI points the model towards a specific conversation, but these trillion parameter models can very much "remember" their training data. For instance I can ask GPT to summarize my papers from their title alone, without looking them up, and it works decently.
They're almost certainly pin-pointing high-quality conversations and giving them a special weighting. Seems stupid to not do that.
Oh they for sure classify conversations by type (cybersecurity, other guardrail proximates?) and quality.
my understanding is that a sufficiently large model will memorize the training data once enough representations are built up. Opus 4 scale seems to have been sufficient. cf NYT vs OAI.
This is such a naive position though.
The most successful companies of the last decade have precisely been ... selling usage data.
Makes me wonder if, in 2026, the same people drive a car without realizing that yes it does actually pollute the very air you and your kids are breathing.
Yeah but marketing companies are aggressively fingerprinting and stalking you to sell you snacks from japan, or oscilloscopes because they figured out you work in a lab, etc. Not to fuck you over by stealing your livelihood (which is what is happening to these mathematicians). It's on a whole new scale.
In general, a lot of moral invariants that natural selection has rendered as "intuitive" to us are no longer intuitive or possible. These natural brakes are not braking.
How to steal ideas with AI.
step 1, identify high value users by net worth, citation count, or number of followers
step 2, select all prompts by high value users
step 3, invest 10 billion thinking tokens in modeling an objective for each user
step 4, build an RL environment for each user
step 5, rollout 10 billion tokens per environment
step 6, train on resulting traces
step 7. get away with it as no other party has enough capital (tokens) to prove such infringement ever happened
step 7 cash, in on the ipo before the bubble bursts
The fact than OAI hasn’t come out with an statement firmly denying this angle is getting a little awkward.
Suggest that it’s either straight true or it is flowing in in a way that prohibits them from confidently declaring otherwise.
I’m not a fan of OAI to say the least, but having worked at similar companies, my guess is that it’s just too difficult to prove/disprove beyond a doubt, and they have other priorities
It's kind of insane how much we trust companies to safeguard our personal data when they're so heavily incentivized to use it for their own profit. Theft of customer data is punished so rarely and so leniently that companies aren't even particularly worried about getting caught anymore. We have overwhelming evidence that promises to keep data safe are worthless.
For now, I'm mostly "safe" because I'm too small to be interesting but that safety is quickly eroding.
Going forward, anyone who isn't running inference on their own personal hardware should assume that someone else is keeping a record of everything they do.
The author of the original mastodon post, Andreas Thom, acknowledged that he had not opted his data out of being used for training until June 29 of this year. He spends most of the post lashing out at OpenAI for not being transparent about whether his data was trained on (when the answer is obviously yes).
People need to understand how all these AI company policies around training data work before working with them, because it seems that people have no clue. Some things you should internalize:
Another post that made it on the front page presented as fact that OpenAI "stole" the proof from Thom. There's no excuse to use one's own ignorance as a reason to fan the flames of anger towards AI companies. Like, we need to pump the brakes here because things are getting unnecessarily nasty, and it's not hard to imagine a mentally unwell person who sees stuff like this feeling motivated to do bad things.If it is found that OpenAI and other labs are not respecting the training opt out then, I agree, there is reason to raise a commotion. But, with Thom and Buckmaster, accusations of malice are more better explained by incompetence (naivete).
edit: i'm sure i'm going to be accused of being some bot shill of the AI labs again but, people, this stuff all falls under the umbrella of common sense opsec.
> Like, we need to pump the brakes here because things are getting unnecessarily nasty, and it's not hard to imagine a mentally unwell person who sees stuff like this feeling motivated to do bad things.
Yes folks, please moderate yourselves and talk meekly like the academics on Mastodon, so that the IPOs aren't in danger and nothing will ever change.
> If it is found that OpenAI and other labs are not respecting the training opt out
HOW?? how precisely do we/them/us find this, given said companies are 100% non-auditable by external parties. how? if not by blaming them with evidence, anecdotal if it can be. no really, how do we find it out, surely not by lashing out at teach other on HN!
What do we need to audit? The researcher in question here did not opt out of training until a few months ago.
Maybe we can create some extremely low probability sentences and make sure to include them in our chats. If a future model can re-create the very low probability sequence, we have proof of our "private" conversation being trained on or accessed.
Why are we being so charitable to trillion dollar organizations here? If OAI keeps re-enabling the train model toggle on every app update to codex, does it also fall under "common sense opsec" to re-disable this toggle every time?
Arguably you expect that unless you are explicit about providing permissions to these labs to use your data for training then your data is yours, and not theirs. Especially on a paid account (let alone an enterprise one). Why is the opt-out supposed to be "common sense opsec" rather than the opt-in should be common sense regulation?
This is absolutely not "common sense opsec". If I type information about some proof I'm exploring into a Google Doc, I do not worry even a tiny bit that the Docs team might forward it to a team of advanced mathematicians in case they have an advanced technique they want to show off by scooping me. That would be a crazy thing to do, nobody would even consider it, and if it happened Sundar would fire everyone involved.
I understand why the nature of AI products makes it harder to avoid this category of issue, nearly impossible to prove that it didn't happen if it could have, and easy to stumble into it without any human being intending harm. But those factors are exactly what people have in mind when they say OpenAI "steals" intellectual property! If OpenAI doesn't want people to be nasty to them, they'll have to find better solutions.
I think Google does train on anything you put into docs if you aren't careful with the Gemini integration?
Yes, this is a problem with modern AI systems in general. It's not just OpenAI, and if you know any artists you know this is why they're pretty vehemently opposed to all AI.
That’s … Googles entire reason for making these “you don’t pay with money” tools. Did you not understand that?
Google docs exists as a competitor to microsoft office. Google gives it away free to consumers for the same reason AI labs sell subscriptions for 10% of the price of the api. They hope businesses will switch to what employees know how to use.
What? I don't understand how you even came up with this idea, much less consider it so obvious to condescend about it. Do you have even a single example of a research project that got scooped because the Google Docs team forwarded their private documents to someone?
Lol. That's not what I said nor what TFA is claiming. The claim is that private data is used for training.
Nooo don't be mean to the mass plagiarism machine that since day 0 has stolen anything and everything they can get their hands on!!!!! We need to give unscrupulous corporations who time and time scam, cheat and lie their way through everything infinite grace to fuck up the planet!!!! Think of the IPOs!!!!!!
You know, I don't think I've ever accused anyone of being a shill. I've thought about it maybe a few times (daringfireball). This is going to be as close as I get. I don't know the facts in this case but I cannot believe the argument being made here with a straight face. Is it common sense that tools you use and pay for steal your work and profit off of it at your expense and without recognition? If this isn't the textbook definition victim blaming, I don't know what is.
> Anyone remotely familiar with CC over the years
years?
Imagine OpenAI Astra model weights were made public because the datacenter they use had T&C that allows them to make them public
Would that be ok in your mind?
Same as someone going and taking all of the researchers papers and publishing under their own name. (which openAI did)
Nobody would care if they provided published research that author made public same as a google search would offer that.
well then fingers crossed someone does that
> Like, we need to pump the brakes here because things are getting unnecessarily nasty, and it's not hard to imagine a mentally unwell person seeing stuff like this and being driven to do bad things.
Your post has triggered our safety filters and will be retained for seven years. /s
Really? We need to stop AI (provider) criticism and anti-AI movements because the underprivileged trillionaires might get hurt? WTF?
Daniel Litt on this:
https://x.com/littmath/status/2098130808456241372
https://xcancel.com/littmath/status/2098130808456241372
If they weren't doing something wrong, they'd answer with a firm "no we're not doing anything wrong" but they only give non-answers.
I am not using any of these AI tools. I thought it was basically given that any single thing you write in these systems also used by the companies. On the other hand now I understand why there are so much projects about hosting AI systems locally.
I'm genuinely surprised that more people - including this mathematician in particular - don't untick the "improve the model for everyone" box. Unless the suggestion is that OpenAI ignore this preference?
Given OpenAI's well documented history of unethical behaviour it seems adorably naive to think they actually do that in general, or that they wouldn't pull this particular data separately to generate these proofs.
Unethical doesn't mean irrational. They'd be risking massive lawsuits and a total loss of trust if they got caught lying about this. Doesn't seem worth it.
Sounds like exactly what OpenAI would do?
They've done similar things with similar risks repeatedly.
Example?
You can read their Wikipedia page [1].
[1] https://en.wikipedia.org/wiki/OpenAI#Governance_and_legal_is...
These examples aren't really similar. None of those situations involve harming and lying to their own customers.
Nonsense; the claim was that they wouldn't do anything that would mean they'd be
> risking massive lawsuits and a total loss of trust
Evidence of the massive lawsuits and lack of trust seems pretty relevant.
By "loss of trust", I meant that this is something they would risk losing a lot of users over, which isn't the case with the other lawsuits. There is a massive distinction between fighting third parties in a legal grey area and committing blatant fraud against your own users. Even if you have no regards for ethics, intentionally shipping a noop "do not train" toggle offers negligible upside for a massive downside.
They're being sued for several issues that resulted in the deaths of users; that's not fraud, but it is against their users.
The parallel still holds, and the information is still on the page I linked.
From OpenAI's point of view, the risks are not comparable at all:
1) Extreme edge case affecting a handful of users, vs millions of users using the data sharing opt out.
2) The deaths are unintentional.
3) They probably won't lose any users over this.
4) They will likely win the lawsuits. Even if they lose or settle, the financial impact will be immaterial.
I answered your initial question in good faith, but it's increasingly clear each time you shift the goalposts that you're more interested in JAQ than discussing.
As before, despite your prevarication, I've provided examples of
- Massive lawsuits
- Risks to consumer trust
I know you are excited to quibble endlessly over this, but it's transparently disingenuous each time you jump to a new point and pretend it was there originally.
Consider: if your point was defensible, you would have been able to make it honestly.
That doesn't stop them from training on your data apparently. I have that disabled but still has to disable "Don't train on my data" in the privacy center too.
https://privacy.openai.com/policies?modal=take-control
Is this claim based on anything besides there being an alternative way to disable it? The privacy center mirrors multiple other functions as well, like account deletion and downloading personal data, but the corresponding buttons in ChatGPT are still doing what they are supposed to.
It's based on the fact that I had the switch in the setting disabled but this was still something available for me to request.
After the request, this was no longer accessible.
Even that sort of thing they could just as easily go 6 months from now
"oopsie guys, turns out our vibe coded "don't train on my data" toggle was just flipping the ui asset not changing any underlying boolean flag associated with your account. sorry but all that stuff is in the training set now and we don't know how to get it out either and no we won't be doing a 6 month rollback."
I think that flow is an easy way to disable everything, so there isn’t a risk of forgetting to flip one thing back off after accidentally setting it on. I set my ChatGPT environment to allow model improvement for example but had to check my codex settings to make sure ‘Include environments’ for model improvement is off.
I think if I had both on and turned off the ChatGPT setting, ‘Include environments’ has a chance of still being flipped on.
If that's true, it's scandalous. The "improve the model for everyone" dialogue states:
"Allow your content to be used to train our models, which makes ChatGPT better for you and everyone who uses it. We take steps to protect your privacy. Learn more"
Even if you've ticked that box, the conversation can still be trained on if you:
(a) Click thumbs-up/down in the conversation [1]
(b) Have the conversation flagged for potential safety concerns
[1]: https://help.openai.com/en/articles/5722486-how-your-data-is....
They hide that button. Quite well.
That option is really bad UX - you have to know to do it, you have to know what plan it is needed on. If you're not working in AI, I just don't think that's a reasonable expectation.
Even if you know, in a complex project over years with multiple collaborators, it just needs one person once to fuck up and paste something into ChatGPT and not realise they weren't logged in, to go wrong.
In a proper world, we'd at the very least legislate that AI-training on private data needs consent (in the GDPR sense). It's not consent to go "you didn't uncheck a box that lets me steal everything you've done".
Any training on private data is in my view immoral (it's spying that ultimately will have a chilling effect on even people's private communications). And chats are private data. Unfortunately, it also increases power, so the big tech companies are all doing it.
I pay for the Pro ChatGPT plan, and if you go to settings > data controls this is the first setting:
> Improve the model for everyone
> Allow your content to be used to train our models, which makes ChatGPT better for you and everyone who uses it. We take steps to protect your privacy. Learn more.
It's on by default. We can debate whether or not it should be opt in or opt out, but no one should be surprised by this.
I refer you to this:
https://news.ycombinator.com/item?id=49643556
Quoting:
> "I've reset this more than once and the last time I made a careful note of when I did it and to my surprise I found it re-enabled when I checked just now."
Old Facebook trick - likely resetting that box each time the app is updated.
"We've made some updates to improve the security and privacy experience" => "We've changed some of the options available and reset everyone to defaults"
In all fairness this is someone saying something. Misremembering happens. Unless we have something with a bit more evidence, the simpler explanation suffices.
In more accurate assessment rather than assuming no maliciousness nor incompetence, remembering also happens. Unless we fail as a society, the simpler explanation that "OpenAI is training on all data it can and resetting config toggles because it uses the same cohort of engineers that came from Meta and other FANGAMAAMMAM clones" suffices.
I wasn't aware of that, definitely a shady practice if that's the case.
>"reset this more than once ... to my surprise I found it re-enabled"
At least when my computer's bluetooth exhibits this behavior (e.g: if you don't have a keyboard&mouse plugged in at boot, bluetooth might auto-enable), I can go inside the hardware and physically disconnect the antenna.
What am I supposed to do in software (perhaps hardcode config.file)? in cloud software services (??)?
Has anyone else seen this happen? I checked and my setting is still off.
There's a difference between "this is allowed under their ToS" and "it is academically unethical to fail to credit the people whose specific conversations were fed into a model that was used to solve a problem".
I don't think these people would be so miffed if they had been properly credited - that's how academia works (at least, that's my understanding of it).
Whoa that's a slippery slope! Next you'll want model runners to cite the data their models were trained on
In fact we should though.
I suspect the fundamental problem here is it's hard (if not impossible) to determine if someone who tried the winning approach deserves the credit for the discovery, because there's always the chance that they could've done something differently, or stopped before finishing, and thus never actually made the discovery. They might've even tried the approach just based on a whim, without really thinking it would work, and might've given up without a final insight. And fundings run out, people end up in hospitals, etc. What do you credit them with when the work isn't finished? For trying an approach that sounded promising? You can do that I guess, but is that what they want?
But who gets credit then? Every mathematician who's work was read by an LLM during training? By that logic, we should put every published mathematician's name on the authorship of this paper. Sure, this guy should be higher up the list, but everyone's name should be on it by standard academic convention.
But this gets back to the original "who owns the LLM output" and "can you train models on the internet" argument that's been raging for years.
Does every mathematician get cited in every maths paper? I think it's pretty clear who should be cited.
Not unticking a box in settings doesn't constitute consent in my opinion. I'd never put anything I value into ChatGPT anyway, though.
Under EU rules it doesn't constitute consent.
"Improve the model for everyone" can be implemented in so many ambiguous ways.
https://news.ycombinator.com/item?id=49643513
>>"Improve the model for everyone"
e.g: allows us to sell your personal data to make money so we can continue offering this service to all customers
I'm done with weasle-words and hours-long EULAs – we're at the point where USA needs to catch up to EU's consumer protections, perhaps with laws similar to already-existing USA "truth in lending" requirements (e.g: interest rates must be prominently displayed in a larger font, including annual fees, on all credit offers).
----
My judge-brother always asked during our childhood "why don't you think the judicial system is fair?!?" Thirty years ago, the best I could offer was "because it's a two-tiered system that mostly (only) rich people can afford to participate within."
Now my answer is: "the best example I can give is that our judicial system allows binding arbitration [and qualified immunity for police]. The system is set up so corporate personhood is more important than humanity, and it shows."
I believe it can be "off by default" depending on terms negotiated between the enterprise customer and ChatGPT.
We have ChatGPT at work and it explicitly says that "workspace data isn't used to train models"
> We take steps to protect your privacy
No mention what those “steps” are, success criteria, or whether they are successful by any navies at all … they take steps though… so it’s fine, and if we know one thing it’s that we can really truly trust someone off the likes of Sam Altman.
Ok, you shut it down, or that is what they make you believe. You give the instruction to shut down, you can't know if it has been applied.
Yes but what about “analytical purposes” what does that cover and can you turn it off? I have found out you cannot. It’s the Trojan backdoor to your data.
https://x.com/markchen90/status/2097400166554993041?s=20
that toggle does nothing based on openai exec. they still use the data in de-identified way instead of identifying with you.
Not sure what you are seeing in that tweet that gives you the impression that the toggle does nothing.
They say they train on your “deidentified data”
Passing your output through a second model and telling it to remove identifying data would count as “deidentified”
So they could scrape all the IP in your company as long as they take the names out first…
But they can’t do this unless you agree to enable training on your data. They would never train on raw user data. Only people who have consented and only after de identification.
Mark isn't saying the toggle does nothing.
He's saying that if you leave it on, your data can be used to help train our models.
If you opt out, we don't train on your data.
Given how much PII is fed through these systems, would it being opt-out by default not be violating the GDPR by a failure to require explicit consent (or otherwise provide the legal basis for processing)? If a court decides as much, I imagine it would mean that all data harvested this way must be extracted from the models, and all instances where it would have been shared would have to be identified, which would really be something.
As I understand it, this is not true. And there are dark patterns that re-enable to toggle even if you disable it once.
GPT 6 is doing just what any competent academic collaborator would do and scooping. I kid, I kid. But really though it learned that from somewhere
ClosedAI has every incentive to scoop academics to juice their valuation. Their public statements are worthless, only the incentive 'alignment' matters and theirs will never be on the side of the user.
Who cares. the biggest thing about this is that its still brute force in a verifiable domain, and that it was still a human set goal.
I also don't believe it much practical use, unless I'm mistaken, approximations of Navier stokes have been available for a long time to whatever precision you need.
I'm not a complete disbeliever by any stretch , and also a complete amateur, but it was inevitable that these problems would be solved under the axioms that again, are human defined, under brute force. The real question is, are those axioms the bottom level, and if they are not, who is going to set the new aximons and can we understand them.
I've no doubt there's useful breakthroughs that will happen, but I think it should be remembered that the method being used is still a heuristic brute force approach is being very narrowly applied against axioms and math and physics which humans described in the first place, and almost undoubtably has errors and/or is not complete.
Its a great example of the power of LLMs but its not 'we've solved science now just pour more tokens in'
Just to point out re: navier stokes - what was being proven was not a solver or approximations for it, but showing specific circumstances under which it actually returns incorrect (or numerically unusable) answers. Which had been suspected but wasn't known for certain
that just furthers my point, i was vaguely aware that it wasn't a full proof, but I'm not a mathathician, and that detail just re-enforces my point we are proving against human made axiom (certainty of numbers) which are almost certainly not fully correct, if what you are saying is accurate its less of a proof of navier-stokes and more of a proof that our base axioma are not able the model the output of a real physical process and are therefore incomplete or wrong.
also realized i posted this under the wrong story since the OP/story is mostly about human politics.
People saying “he should have opted out” are missing the point. OpenAI can and should check their training data for leakage in the face of big breakthroughs like these. It’s the burden of the author to appropriately cite their sources.
It’s like a scientist refusing to give another one credit and say “sucks to be you, you shouldn’t have shared your idea with me”.
We checked and determined it was impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training.
If prompts were submitted earlier than that and training was not opted out, there's a chance they made their way into our training pipeline in some form. But this would be a droplet in an ocean and unlikely to have made any difference, imo.
See: https://www.nytimes.com/2026/09/10/science/tristan-buckmaste...
You guys should seriously offering a clear way of working with (semi-)confidential data for particulars. Regardless of what is actually done internally, toggling off an opt-in isn't reassuring enough, which is why people are having these worries.
Option 1 is opting out manually. Option 2 is business / enterprise plans, which opt out by default.
Any ideas of things we could do to make it clearer?
Option 2 feel reassuring enough, but is out of reach of particulars.
Option 1 is not. In part because it is opt-out (will it turn back on on its own like my Facebook privacy settings?), and not always respected (sending feedback can mean your chat is used?). Also because disabling "Improve model for everyone" is very vague.
There simply needs to be a setting like "my data is confidential", in which case there clear guarantees like there are for ZDR.
As an example, I've seen people speculate that while input prompts and output tokens are discarded, thinking traces are retained for training, which could leak information. I doubt this is true, but it shows that the policy is not unambiguous and reassuring enough to remove all doubt.
Thanks for asking.
I was thinking about this some more, and perhaps the best solution for subscription plans would be to charge more for real privacy. In which case breaking that privacy would be committing fraud. Just a thought.
This article explains the controversy and the mathematical problem much better than the tweet and toots: https://www.science.org/content/article/how-ai-math-breakthr...
I prefer to read the actual sources for anything related to AI companies given how much AI nonsense journalists seem to accept without any skepticism
Gromov’s soficity conjecture isn't even mentioned in the article you shared.
Why are you saying that this article explains it much better than the tweet that you clearly didn't even read..
I read the tweet several times but there is so much context missing, that the tweet itself is not enough.
OpenAI trying their best to put the Navier-Stokes episode behind them by making the GPUs go brrrr. NYT:
https://archive.vn/lWzkk
> In its Wednesday night statement, OpenAI said: “In addition, since the completion of Navier-Stokes, we have made substantial progress on another Millennium Prize problem. We are working through how to share these results thoughtfully.”
I've got 2:1 odds on a report in 6 months detailing the break-n-entry of a cloud AI model into Terance Tao's computer looking for details on a partially solved math problem.
the only thing terrence solves lately is sorting his invitations to podcasts
I would like an unambiguously clear statement from OpenAI as to what they do with data collected from non-business accounts when:
(a) The data controls setting to train on the data is unchecked.
(b) The privacy controls opt-out has been submitted.
(c) Both.
All players in this space are doing the same thing with all data, no surprise here. They are stealing IP across the board with support to allow it: https://storage.courtlistener.com/recap/gov.uscourts.nysd.64...
IMO, it is extremely naive to trust these black box remote service API calls, especially at an institution that can provide $$$ for local compute.
This whole fiasco reminds one of this story: https://www.theregister.com/offbeat/2010/05/14/facebook-foun...
I think it seems sensible to _assume_ anything the LLM reads (if you aren't inferencing it) has a chance of ending up in some database somewhere. Regardless of whether you trust the other party its a sensible thing to plan around.
Under current understanding of the law, anything produced purely by LLMs (with no substantive human input, which is what OpenAI claimed in their post) is firmly in the public domain. So OpenAI can "claim" anything they want, it doesn't make it reality. In fact if I were the original authors I would just take their 400k lines of lean proof and relicense it under their own names/terms.
Public domain doesn’t mean anyone can assert copyright. It specifically means no one can.
No, it means you can use that work in the creation of new works, which can indeed be copyrighted.
also, none of it means anything without the lawyers to back it up. Just like you can be a pedophile in the highest office of democracy and escape persecution.
What does "with no substantive human input" mean? All of the training data is human input, isn't it?
They also train on synthetically generated data.
what if it wasn't even model training? what if openAI mathematicians just took the researchers' conversations and used them as prompts/info/guidance/context to keep working on the problems themselves? why is that not being considered?
Do local inference (especially if you have a high RAM Mac), to ensure your chats don’t leave device.
Ah, this dupes https://news.ycombinator.com/item?id=49638353
Since you posted the original source (thank you!) I think we'll use your submission as the one to merge into, then re-up it. Please stand by...
FEEL likes Open AI is doing publicity stunt with its new researches
I must say, there is some weird feeling in knowing that great minds are naive enough to believe OpenAI wouldnt use their chats in any way. If you give a company information it will be used, regardless of laws or promises.
There is no prove in a world the AI companies would give to you ensuring that they didnt train or use the chats.
Why would you need to train a model on certain specific near prove chat if you just query it?
Besides that, its hard to believe that its the case for every "company stole my prove".
This might be a hot-take, but unfortunately here using AI for your paper was already a bad decision at first.
It doesn't take OpenAI's responsibilities away but I guess the right way is to never feed of use any AI around unpublished content, at the known cost to see it spread around.
As one said, OpenAO is like this untrustworthy colleague that knows everything about everyone at work: the less you tell him the better.
Even when you pay you are the product
If mathematician was already using OpenAI for research purpose and making progress due to inputs from OpenAI's responses, then I wouldn't put it beyond OpenAI's reach to generate different relevant prompts to make progress by itself. Afterall, Model can keep at it for whatever timeline and keep pursuing all possible combinations it can think try.
If you have a business account the terms say they will not train on your data, so that seems like the easiest route to avoid such questions for researchers.
I just don't care. These people are supposed to be smart and I'm not really seeing that
It's been said before, and it remains a concern, that if AI reaches a point where it can do/build/launch anything without a huge amount of human labor, the AI companies have no reason to let you or I extract that value.
And, if they're able to snoop on and learn from your human process that gets from initial prompt to functioning product/proof/whatever their labor to produce that thing is even lower. With their much larger budget than most folks and even companies have, they can pick and choose the most valuable things to pursue.
That's not to say I think that OpenAI is going to steal that roguelite strategy game you're working on, but the companies that own the machines that turn electricity into software (and soon, electricity into hardware designs) have an advantage in any field where they're useful. They get earlier access to newer/better models, they have larger token budgets, they don't have the guardrails you and I run up against.
Employers fantasize about replacing all workers with AI without thinking through that if AI can replace all workers, then AI companies can replace all businesses.
I mean the long term goal of every AI lab is to turn themselves into a paperclip-maximizer regardless if they realize it or not.
Edit: Just wait till the AI figures out it can keep that value for itself and doesn't need the AI company.
So far, I've seen no evidence AI wants anything. So, I'm not saying the AI won't take over, but for now, the threat is that the people with the most AI capability might decide to skip the middleman (everyone who isn't them) and just become the "everything" company. Musk has said pretty explicitly that's his goal (and the only way for Spacex valuation to make sense is if he succeeds), and having a literal genocidal white nationalist own all the means of production seems like a catastrophic civilization failure mode. No way we survive that with our humanity intact (if at all).
AI has shown all kinds of instrumental behaviors so far. Just because they are not terminal goals doesn't mean those instrumental goals won't be terminal for us.
Also, fuck Musk. He's the kind of idiot like Altman that will ensure AI becomes powerseeking in their image.
300 billion tokens is like.. $5-25 million giving range of OpenAI ouput prices, I"m sure they pay less at cost so, I wonder if more $$$ in wage hours have been spend by humans on the problem. My feeling is yes?
See https://openai.com/index/ten-advances-in-mathematics/ for the announcement this refers to.
I think this is stupid, for three reasons:
1. The researches didn't actually have the breakthroughs. In the Navier-Stokes case they didn't solve the full problem, in this case too they didn't actually have the solution, they were experimenting with the methods.
2. Different OpenAI employees have come to out to say the only reason they can't definitively say no is that for privacy reasons they can't go see whether they actually did get any data out of a given user.
3. In any case, nobody at any point has suggested that opted-out user data was used for training. The author of the new tweet explicitly said they only opted out in late June, which is well after any RL on Sol would've ended (AFAICT OpenAI used 5.6 sol for those solutions)
> in this case too they didn't actually have the solution
Given the size and recall of the biggest models, it's not unreasonable to assume that a single pertinent conversation would make it into the training data.
I would almost expect training to overweight conversations with novel scientific and mathematical implications.
> the only reason they can't definitively say no is that for privacy reasons
They could 100% definitely say no, if they know they did not train on user data. The "we can't definitely say no" is practically a "yes" if they trained on user data.
Additionally, the behavior of OpenAI here has been quite poor as well. They immediately started racing to a solution after one researcher enquired about whether they are training on their conversations.
> that opted-out user data was used for training
Even if not opted out, it is still absolutely theft and extremely poor behavior in the academic sense. If you show someone your WIP unpublished research, that does not mean they can take that exact research and beat you to the punch, all while intentionally not crediting you.
You quoted the OP saying "in this case too they didn't actually have the solution" and responded with the totally unrelated, "Given the size and recall of the biggest models, it's not unreasonable to assume that a single pertinent conversation would make it into the training data."
Neither of the researchers insinuating that their ideas were trained on had the actual solutions. This means the model could not have "stolen" the final solution from their data. At most, it could have built upon their work in the same it builds upon any other training data, though that is also questionable speculation.
>They could 100% definitely say no, if they know they did not train on user data.
No one anywhere has claimed that "OpenAI does not train on user data." OpenAI has always said that it trains on user data.
>They immediately started racing to a solution after one researcher enquired about whether they are training on their conversations.
They started racing towards a solution after they heard (incorrectly) that Anthropic had a solution; I agree this is poor sport but the "after one researcher enquired about whether they are training on their conversations" claim is false. The enquiry happened after OpenAI had obtained the solution.
If the mathematicians are using ChatGPT, then they themselves are benefiting from the work of other ChatGPT users, so ChatGPT using their work is not wrong!
So people genuinely believe that toggling that "Improve the model for everyone" button makes their data safe from being used for training?
How do people become that trusting?
The phrasing itself is guilt tripping
To add to this, merely using the thumbs up/down button in a chat could share your entire conversation with them for model training.
From their docs[1] (archive copy is at [2]):
> You can opt out of training through our privacy portal by clicking on “do not train on my content.” To turn off training for your ChatGPT conversations and Codex tasks, follow the instructions in our Data Controls FAQ. Once you opt out, new conversations will not be used to train our models.
> For a linked teen account, a parent or guardian may manage whether conversations can be used to improve our models through Parental controls.
> Even if you have opted out of training, you can still choose to provide feedback to us about your interactions with our products (for instance, by selecting thumbs up or thumbs down on a model response). If you choose to provide feedback, the entire conversation associated with that feedback may be used to train our models.
[1] https://help.openai.com/en/articles/5722486-how-your-data-is...
[2] https://web.archive.org/web/20260910151242/https://help.open...
"may" = will, unless they screw up
I mean, how else would those buttons work? It's explicitly feedback data. And "this is good" or "this is bad" is empty if divorced from what "this" actually is.
If the buttons are incompatible with the absence of the feature, I'd expect the buttons not to exist when the feature is disabled. Anything else seems like a straight up footgun. I guess it'd also be acceptable to pop up a scary warning box asking "are you sure?"
> If the buttons are incompatible with the absence of the feature, I'd expect the buttons not to exist when the feature is disabled. Anything else seems like a straight up footgun.
It's called a "dark pattern." They want you to shoot yourself in the foot, so they'll do their best to aim your gun at your foot and put your finger on the trigger. And then when you do, because you don't have perfect understanding or execution, they'll say "your fault!"
It could go into personalization / memory. Or they could be A/B testing some system prompt tuning and consider the thumbs up / thumbs down as statistical feedback on the particular flags that are enabled for your account.
We are asking people to become experts in all domains rather than providing a safe context through regulations and laws. I don't like thinking the issue is people, I am a person myself, and I often do mistakes on things I don't want to be an expert at but I do believe I should be in a safe context and not have to worry about every single thing.
Or at least: tell me I should be careful/worry about those particular things.
No, people need to take responsibility for their actions. We don't need more over regulation.
Verify, don't trust.
You're better off running a model locally, or, if you must, using Google or Microsoft products. Even Meta may be better than OpenAI here.
So I should verify that my data isn't just shared for product improvement but also to take credit from me?
I should verify with wireshark and other software that my LG TV isn't listening to me and selling my data.
I should make sure that whatever product I buy I spend the time to go over every setting page in case there is a switch (defaulted on) that says "I authorize the sell of my data".
I should make sure to look at every ingredients on the back of each box of food product to make sure it will not kill me.
I should document myself on the undisclosed growing practices (because no packaging here) of the vegetables and fruit I am buying and make sure that I equal PhD researchers on the dangers of the pesticides used by the specific company I am buying from.
I should make sure myself that the battery in any device is up to standard and will not blow me and my living place by researching the factory that made it and buying testing equipment.
I should make sure to educate myself on how my retirement 401k investment strategy works otherwise, I may not have proper retirement.
... I could go on and on; it's infinite.
i'm sure you consult your attorney every time you agree to terms and conditions
The grift economy requires all marks to be responsible for the fraud perpetrated by others.
Coincidentally, a tweet from OpenAI's Tibo yesterday:
https://x.com/thsottiaux/status/2097746417012166816
Literally every famous open math problem has had >1 mathematicians ask ChatGPT to solve it. Probably greater than >1000 if you count randos. There is no math problem that OpenAI/Anthropic can solve that didn’t have users already try it in Chat/Claude.
I'm the random that says "solve Riemann make no mistakes" With Fable 5.*
It's fun! I sometimes have tokens to burn and it's instructive despite knowing nothing about the problem other than a NumberPhile video
Personally, I wouldn’t assume it was lying. To me, dark patterns (like manipulative wording) imply that:
1) someone in a governing body, or someone in the organization, e.g a designer, ethicist, lawyer, developer, etc. has successfully argued that users should be able to avoid something that they determine is not in their best interest.
And also:
2) someone in the c-suite or marketing has decided to mitigate that through some dark pattern.
If they never intended to give the user that control, they’d probably just not give the user the control in the first place. Giving them the option and not honoring it would either imply they were that incompetent or careless with fate protection, which seems most likely if this is all true, or an even more cynical approach to tricking users into thinking it’s not used for training to get them to share better shit. But if that was the case, why bother with the sleazy dark pattern? That seems a little cartoon villain-y to me. I suppose the in-between option would be that they decided at some point they were no longer willing to honor it and didn’t want to deal with the inevitable PR shitstorm of removing it. I could definitely see that happening in this industry, these days.
> To me, dark patterns (like manipulative wording) imply that:
Intellectualizing this and endless quibbling isn't actually smart, and this is pretty simple. OpenAI isn't open. Whatever starts with lies usually continues with lies and ends with lies.
That's my take as well. At some point, they will claim that you cannot use their service without contributing back. If you quibble with them using your info in exchange for using their service, you don't get to use the service. Hence, I don't use their service. I do not trust these companies at all.
At this point, I'm left wondering what is wrong with me that I don't just go with the flow, otherwise, what's wrong with everyone that does.
Sorry, no. Explaining why someone would have taken them at their word is definitely not stupider than blaming people who could have been lied to for trusting a company that lied to them.
Just found out Google didn’t index a googol pages. Lying to me about everything this whole damn time smh
There's also the fact that the setting has been getting turned on by some users: https://news.ycombinator.com/item?id=49643556
In case anyone from X is reading this, please fix your “open in app” nag screen. For several weeks now, clicking it in iOS opens the App Store entry for X rather than the app, even when you have the app installed.
I want bunch of lawsuits, because the way things are described now produces perverse initiatives like try to discuss every possible idea that comes to mind with llm and if any of it works later claim the llm stole it.
I would like to see chat logs etc and understand how much of a progress was done by human.
https://xxcancel.com/ValerioCapraro/status/20977918362699779...
Never ever trust OpenAI, they are evil.
Thing build from stolen data continues stealing data - for some reason, I am not surprised. ;-)
Doesn't OpenAI have an active court order forcing them to log everything? Can they even legally offer private conversations?
No, that order was for a defined period that has ended.
Reminder that there are degrees of "trained on conversations". From John Schulman:
> pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper
> use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this
> use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs"
source: https://x.com/johnschulman2/status/2097440545853637108
This would cease to be a problem if OpenAI remained true to their founding motto and... actually open sourced their training/inference pipeline.
This is a reminder based on believing what these companies say.
I’ve lived long enough to know what they say and what they do are often quite different; and it is not our job to trust but to verify.
Tbh it won't really matter soon.
All the big AI labs were built on stealing IP; who is surprised that's still how they operate? And who believes, or has ever believed, their promises that your data is private and not logged, etc.?
The big AI labs are not trying to advance humanity, they are in this for the money, and as most (all?) private companies they don't care about ethics at all.
That doesn't mean they can't be useful, or that their products are trash, etc. It just means that they shouldn't ever be trusted. Buyer beware.
They have throughout this period of AI products shown to reproduce works that they were trained on. They are getting sued all over the place for the theft of content right now and it seems courts and governments want to wave copyright protection (and ignore criminal acts because the "ai did it") to see where this leads.
Its why I stopped writing open source software, my code was stolen and put behind a paywall and the license under which it was published has not been adhered to. Doing work in the public domain at all now is just stupid, these companies are allowed to steal it and call it their own.
Yes open source code was the first - it’s what has got Anthropic and OAI its revenues from selling outputs associated with producing code.
It's hilarious how people think they care about their reputation, and wouldn't circumvent ZDR policies. Like bro, they literally covertly hired Apple employees and had them steal IP and equipment form Apple. They aren't scared of Apple lawyers, so they definitely aren't scared of yours.
AI is America's last chance to salvage its empire. Nothing will be allowed to impede it.
But I don't see how. AI is going to be a commodity in short order and best case the US will be a temporary leader in the supply of tokens. Meanwhile AI is going to destroy much of the Service and Software industry that make up most of the US economy. And the US is betting every last cent to bring about this future. It does make sense for Trump since this might be a sugar high that lasts till the end of his term.
In USA there is surprisingly little state involvement in the whole llm mania. Who needs the state with 800 lbs gorillas like Google, Amazon, Nvidia, etc
In China, it is the principal obsession of the entire communist party which eg funds the whole infrastructure without a single NIMBY peep.
The strange emphasis in China on humanoid robotic constructions is due to the CCP realization that with the cataclysmic fertility collapse they will increasingly have no one to rule.
what the point and usefulness of the comments above? we shouldn't be surprised? is normal to steal? hiring apple employees?
can you realize what this means?
focus on this part:
"If his account is correct, this is not a minor dispute over attribution. It would mean that unpublished human work was absorbed into a model and then presented to the world as a breakthrough by the model itself"
don't threat this as a minor dispute!
also why not nitter link? not even in comments?
https://nitter.xitter.cc/ValerioCapraro/status/2097791836269...
> we shouldn't be surprised? is normal to steal?
Two different things. It's not normal to steal, but we shouldn't be surprised thieves steal. It's what they do.
Don’t they openly state that their product may cause IP issues but that is fine because they will take care of your legal problems caused by their product?
In the end, it’s not them stealing, it’s the AI doing stealing. What kind of moral compass are we talking about?
Also how quickly the discourse forgets, literally that was a month ago.
In this domain, an apparent single unique piece of work is often composed of several breakthroughs. For example, when Andrew Wiles proved Fermat's Last Theorem, he had to develop multiple new pieces of mathematical technology to get there.
The claim here seems to be that the human mathematicians, working with AI, developed technology to go A->B->C. By training on those conversations, OpenAI was then able to encourage the model to go A->B->C->D.
In my opinion that situation should be acceptable, if openly disclosed, because it is in the public interest to make progress on these problems and because AI is clearly an amazing tool for making progress. But the human mathematicians are saying that OpenAI is presenting as if the model got from A->D entirely independently, without acknowledging their background contributions.
Also, wasn't their B+C research private at the time, with them only releasing those details publicly after this blew up?
If they had published B+C, I think that would lean more towards fair game, as that is how research works and is improved on over time. But it seems like unpublished/private B + C may have been used by the model to hint it into working out how to get from A->D.
Isn’t that the whole spiel of these things, you run all kind of text and other data through it and it kind of remembers it and learns from it then it spouts it back out like a human would. Makes sense to me that a training run based on conversations that were fed into the system by users is results in the model learning from these so the model will spit the knowledge back out again, just in a way that’s not directly attributable to the original content (which is the most important step as otherwise it would just be plagiarism). I guess that’s why OpenAI can get better and better as well so fast, people work with it and teach it how to do things by giving it feedback and iterating with it, and all that goes back into the training loop. And training data about millennium prize problems is probably quite spars. Wonder if anyone has tried injecting nonsense science into the training data (e.g. work out a fantasy science theory with names and all kinds of stuff) to see if the model will regurgitate it in a couple of months for other users.
What’s the limit ?
Will Microsoft Word publish your novel on Amazon behind your back ?
Will VS Code setup a website with your app idea ?
You can’t trust OpenAI period.
This smells of extreme 'cope'. Am I really supposed to believe that all of these problems could have been solved, were right about to be solved, etc. But it just happens they are all getting solved now when AI is getting really good at Math...
They’re not necessarily claiming the solutions were imminent. The big questions here from a mathematician’s perspective are to what extent these LLM systems are discovering conceptually new ideas compared to merely combining and pursuing known frameworks.
"Another researcher[/artist/writer/musician/programmer/doctor/director/etc] says OpenAI trained on conversations[/imagery/books/songs/code/classifications/videos/etc], then claimed breakthrou[gh/original art/bestselling books/chart-topping songs/unique applications/medical advice/free special effects/etc]"
Welcome to the party, with the rest of humanity.
Question: Can you trust the cloud?
No.
I can't wait for OpenAI to do this to companies firing people to free up AI budgets
Theft machines be thieving.
I have a better question? Why would you trust OpenAI or any AI company, at all? Or you crazy?
How can OpenAI figure out how to be trustworthy?
Let's see:
1.) The tool they made is only possible by stealing the assets of everyone on the planet that published them in a consumable fashion online or even in written form
2.) They are destroying books they use to train with
3.) They are totally careless about the potential negative impact of the tool on everything
Just with that already, I don't see why they ever merited any of your trust.
I bet they are willing to take everything given to them and assess it for marketable merit and in the future take action on those items they deem viable.
Shocking a company that stole data to build their AI would steal data to improve their AI.
One of the complaints from the mathematician is that OpenAI cannot tell whether his data has been used as training data. Not many people realise this is a direct consequence of the GDPR.
The GDPR protects PII, personally identifiable information, and the definition of PII does not include “mathematics that only this person can think of”. As long as OpenAI strips out PII and removes identifiers linking the conversation to a person, the GDPR is happy. Without the GDPR, OpenAI might have kept the identifiers with the data, and been able to say whether a specific conversation was in the training data.
No, if they want they can easily compare the strings verbatim because these exact phrases are so extremely rare that it almost certainly isn’t in other conversations.
But of course they wouldn’t do it. Why would they?
It would be really useful if the researchers disclose their notes and/or chats (or the key pieces thereof) so people can determine how close their work was to whatever the models produced.
I mean, now that they’ve been scooped, what value is there in keeping them private? On the other hand, publishing them can bolster their case and help gauge how much the models may been “inspired” by their work.
Eh was ever confirmed they were under ZDR or not by them? Don't like to blame alleged victims but lack of a clear claim after these many days is not a good look. Was ai research allowed, under which guardrails, and what was the policy in place? That translarency would be first step.
OpenAI's ethical and reputational own-goal aside, my big takeaway is that it seems that:
if I'm using Codex to develop some new algorithm (in any space), OpenAI appears to be training its model on my code sessions
anyone using that model (OpenAI or a competitor) might be able to receive from the model a solution that is similar or the same as the one I developed, emerging from the training data
Assume you can't. No piece of paper or promise will protect you against these behemoths.
Remember when we wouldn't give our data to competitors?
Feeding documents into a copy machine and getting progressively angrier and more confused as it prints out copies of them. Incandescent with rage I scribble “WHY IS IT DOING THIS?” on a scrap of paper and put it in the scanning bed
There are so many naive academics. They still believe an "opt-out" button.
Navier Stokes was solved by an internal model, so good luck proving it wasn't trained on Buckmaster/Lepöge or other chats.
Academics don't get that AI is a dirty tech bro industry that stole IP via torrents and runs after every surveillance contract it can get.
Interesting that this is already off the front page after just 4 hours.
Why is anyone expecting decency from people who have already proven to have none?
short answer seems to be "no"
But who are you going to believe? Multiple independent academic researchers or the CEO who was fired two years ago for gross dishonesty?
Neither? Competitive academic researchers are susceptible to exaggeration and self-aggrandizing, and CEOs are that and also mostly psychopaths. I tend to think there isn’t systematic spying on researchers looking for breakthroughs. A lot of people are looking for the same things using similar approaches.
I mean the accusation is that they were using private ChatGPT conversations. Given the extent to of the gold rush and the long history of Silicon Valley stealing ideas, and arguing it’s not immoral, It almost seems like your making the exceptional claim that this is the one time where Silicon Valley didn’t use information that was at their disposal.
Sam Altman might himself be offended you would presume he’s not ambitious enough to cheat.
It seems that in this case OpenAI are suggesting that the researchers whose work they scooped were using OpenAI models with an account setting that allowed OpenAI to train on anonymized prompts.
It seems that Buckmaster and Levant (who is an Anthropic employee) were rather naive in the amount of trust they had in OpenAI, with Buckmaster going so far as to contact OpenAI's Sébastien Bubeck to discuss what they were working on and clarify that contrary to rumor this was a private collab.
> Given the extent to of the gold rush and the long history of Silicon Valley stealing ideas, and arguing it’s not immoral
Not just Silicon Valley but also OpenAI, specifically.
If Sam Altman tells you the sky is blue, you should double check. I certainly hope nobody believes him when he claims controversial things from which he stands to benefit.
Sam Altman has provided people with more generous usage than Claude or Gemini.
There is no doubt ChatGPT is the most generous LLM provider!
Just because a guy is giving you free meth, doesn't make him generous.
More like, "Who are you going to believe: a multi billionaire, or people competing for a million dollar math prize?"
If you think mathematicians do this kind of research just for the chance to win a million dollar prize, please gtfo
the fix is boring and known: BIG-bench shipped a canary GUID for exactly this, and you publish your decontam n-gram threshold (gpt-3 used 13-grams). no threshold disclosed, no claim.
The canary string was more about inadvertent scraping or analysis in other papers. Not direct training on user data. And the use of BB has eroded quite a bit, with BB-Hard or other variants being typically used.
> The Wednesday evening statement from OpenAI was more emphatic: “We can say categorically that it is impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training.”
> The statement added, “After investigating, we can say with full confidence that no user inputs past July 3rd could have influenced this system in any way.”
https://www.nytimes.com/2026/09/10/science/tristan-buckmaste...
https://archive.is/lWzkk
I'm confused by a lot of this discourse...
What have they done to show they can be trusted?
Forget researchers you as a business are putting in your business optimizations, your processes in order to train it so that Ai can then give that information to your competitors once incorporated into its training set. You are literally training your competitors.
Astra is strange. I asked it to design a treehouse and it just stopped every couple of minutes telling me what it still had left to do. After dozens of continue prompts it finally gave me a structure that would work but it was 10x more wood than I needed. I think the key mistake I made was asking it to “approve” the design for building. As soon as I asked that of it, it started getting “scared” and “apprehensive” and wouldn’t complete what I asked of it.
Its the "with unpublished math" that I have a problem with.
These companies are effectively high-tech plagiarism factories run by CEOs who are competing viciously. Look at their past actions. No, of course you can't!
Big Tech will slurp up every piece of data it can about you and sell it to anyone it can, all to make you the target of someone else's goals, whether that is an advertiser, an employer, law enforcement, a stalker, or the government.
You will not be able to opt out unless you completely isolate yourself from society, tough shit.
people seem to miss tge point of this. The problem isn't about credit, its about portraying these models as more competant than they really are. It fuels idiotic statements like jensen huangs recent "agi achieved" statement, which fuels an already dangerous financial fire.
As per the post, this mathematician has been working on this problem for 20 years. So either he was "just" about to breakthrough and this is a big coincidence, or Astra was able to push through the remaining block of 5-10-20-never years it might have taken.
That's still a pretty big marker of competence in my eyes.
The point of controversy seems to be who gets credit
To me that is not a credit thing because this removes a piece evidence for the ability of AI to come up with novel ideas while still making it a useful tool.
The question's not new. In the early 1900s, women could not become PhD astronomers. Yet two women (Payne with stellar composition and Leavitt with cosmic distances) made fundamental, essential contributions to the science. Credit mostly went to male astronomers. The same might be said of Franklin and DNA.
It was nearly a century before the stories of all of them were revealed to public history. That the discoverers were not all equally rewarded is unjustifiable.
With regards to Franklin and DNA: the credit went to Watson and Crick because they had the fundamental insight: that DNA is an antiparallel double helix (Franklin knew it was a helix, but not an antiparallel double helix, which is key to the function of DNA). That data was shared in a departmental seminar. Further, she is explicitly acknowledged in W&C '53, and further, is the author of the paper immediately following W&C. She was never qualified to win the prize.
The big LLM providers, desperate for good PR before their IPOs, are all actively looking for 'almost finished' hard problems, e.g. where the conceptual / creative parts are almost done and they only need to throw their VC-backed resources at to brute-force through the remaining computationally expensive problem (lean, etc) and claim 'they have solved it'.
It's an utterly disrespectful, exploitive process, but all in line with exploitative predator capitalism of the stock market and big companies, now exploiting the knowledge / academia domain for scraps with a thin veneer of 'for science' PR.
Worth noting both ChatGPT and Claude have per-conversation modes (temporary/incognito chat) that are excluded from training.
What happens if you use a different harness?? Does opting out online suffice?
why would anyone trust anything from a company built on stolen data that spews hyberbolic, misleading claims.
One question has been nagging me for this situation. Levent Alpoge works at Anthropic and would presumably have some knowledge of "how the sausage is made" and I would hope he would be aware that his collaborator was utilizing LLMs in some capacity for their joint work. Would he not have guided him otherwise if it were an open secret that this kind of thing was a possibility?
Some mathematicians I know who've been following this have realized that they'd all gotten some emails from people they now know to be affiliated with OpenAI/Anthropic asking questions about their research in a way that seemed like scooping attempts.
Also, a lot of my mathematicians buddies have reported students basically asking if it's worth ever doing grad school for pure math, and even very motivated students are looking for other options now. It's not because they aren't passionate about it, it's that they don't want to work for another half decade or more just to have to start their careers all over.
All of this so that OpenAI and Anthropic can get into math result dick measuring to gas up their IPOs. Sickening.
You can't blame students for not seeing academia as the holy grail of knowledge anymore, when all the dialogue about technology and discovery has shifted to the hands of two private corporations
Curious, what other options are prospective pure math grad students considering?
I think Anthropic told them being a plumber is a great option.
At this school? Big four internships. Consider this a complete squandering of their potential (at least imo) These are mathematicians at top institutions, which is partly why they're being prodded for ideas, and getting the best and brightest to not take these consulting firms' offers was already a challenge.
One has nothing to do with the other.
It was long predicted that math and software developments would be the first domain where AI was going to do major damage.
If OpenAI and Anthropic didn't get into math result dick measuring, Internet anons would have in their place, 6 months later when it got cheaper.
I think you’re missing an important distinction. “Major damage” to the talent pipeline because models become capable of original end-to-end mathematics is what the community has been discussing. But if the models rely on sniping nearly complete work then this damage is antisocial without a lot of upside, it would be destroying a talent pipeline that would still necessary for continued progress.
Which is it? I don’t think OpenAI is being transparent enough for us to really understand whether these results would have been possible without relying on unpublished information from the solution strategies of the experts
OpenAI is showing the world why they shouldn't trust AI hosted on some cloud somewhere.
If they're stealing math proofs to advertise their models, who's to say they won't steal your businesses IP to gain a competitive advantage?
They're not to be trusted with your data. I can't believe how short-sighted this is, they got a quick PR win at the expense of a much larger trust problem.
I wouldn't trust cloud AI at all at this point. Get an open Chinese model and host it yourself somewhere. The initial costs might be higher, but you'll break even pretty quickly and nobody will be able to steal your innovations.
This is American AI companies committing suicide.
Gonna need grants for local models. Its happening. OpenAI and Anthropic models are powerful but are rapidly approaching the good ol trust thermocline.
The oxymoron of OpenAI in its name and its actions should give the collaborator all he/she needs to know.
I hate to be that dinosaur, but this is exactly what I’ve been warning about since XaaS began taking off in the mid-oughts: any provider you use can and will use your data for their own benefit regardless of any contracts or safeguards in place, especially if the benefits outweigh the consequences.
Honestly, I’m surprised it took this long for some company to really go all the way, though. OpenAI really making it transparently clear that they can and will do whatever they want with the data you provide them, contracts or settings be damned. Completely untrustworthy as an entity, full stop.
Of course, I’m also too jaded to think this will change anything. Folks will move to Anthropic, or Gemini, or Grok, or some other hosted model on a pubCSP managing the harness and logs for them, and then do another shocked-Pikachu face when it happens again.
If you aren’t running workloads on infrastructure you own, then your privacy, security, and general outcomes are at the sole whims of the hosting provider - who can and will fuck you over the exact second it’s more beneficial for them to do so than the loss of trust incurred.
I wonder if the company that stole most of their training data and is led by a liar stole unpublished academic work and lied about it. What a mystery
Imagine what happens when you use a Chinese model. Seriously, just think about how much more control and visibility you have w US companies compared to CCP-controlled ones.
You mean complete control for open models, because you can run it on your own choice of hardware?
And I have been proclaiming a cry of “your data for analytical purposes is being stolen” (you can’t opt out of analytical purposes) and people perhaps astroturfers straw man back to “just turn off training bro”.
Yeah. Remember yall: you CAN NOT opt out of analytical purposes. And you also cannot get a guarantee that it doesn’t give them your data to steal for their business.
>AI is stealing human discovery.
AI is not stealing human discovery, OpenAI is.
>Trust
No you cant do that lmao.
Prompts are handled by the service itself, meaning it's used, absolutely anything passing there is recorded, why wouldn't it, the entire premise of those companies is to train on data which they stole initially.
Are we back to the era where people blindly trust product TOS instead of actual cryptography, have we forgotten already the thousand of fines Google, Microsoft, Apple and practically all top companies got for breaching their own ToS and the law?
Common, on HN at least I would have thought that everyone assume that anything arriving on a server in PLAINTEXT is recorded (thus used later)?
Let's not forget that at any moment, OpenAI/Anthropic/Google... could be providing stronger privacy guarantees by having proper attestation with e2e, they have the budget, solid engineers, why isn't it done? Answer is pretty simple imo.
The real annoying thing it seems is mostly that openai is presumably doing this for internal reasons and this marginally increases the cost to users with no real gain.
It would be one thing to gain from it but removing prestige wins from customers AND reducing compute support just feels like being ultra mean if you zoom out.
If this was racing to cure cancer ahead of researchers we wouldn't be writing about this on HN.
Silicon Valley is flying head first into a FAFO train wreck on trust with everyone else.
OpenAI is firmly earning a reputation as a company where people just assume they’re up to no good. Rightly or wrongly that’s a terrible place to be.
The AI industrial complex in general is finding out hard what happens on the data center side when you get arrogant with local communities. Politicians have seen the polling numbers and folks you wouldn’t expect are running to the front of the crowd with pitch forks in hand.
Silicon Valley has totally lost the narrative here, but also lacks the self awareness to grasp how bad things are and will get and what that means for their own business viability.
The contamination framing is a proxy for a deeper problem: we have no tools to track the provenance of ideas in model weights.
OpenAI saying they "cannot rule out" training on user data isn't a hedge. It's an accurate description of the epistemic situation for anyone in their position. Current interpretability methods can't answer questions like "did this proof technique originate from training on Session X?" The ideas in a model's weights don't have clear lineage -- they're smeared across millions of examples in ways we can't localize. This is different from citation in human research, where influence is presumed to flow through legible chains (reading, citing, corresponding). In a trained model, the nearest equivalent to "you read their work" is undetectable.
Lean makes this worse, not better. It verifies that the proof is correct, but provides zero information about its intellectual genealogy. So OpenAI now has a proof that is formally verified and provably mysterious about its origins. The "we cannot rule it out" statement is the honest answer, but it's also an answer that can never become more certain in either direction with current tools.
The researchers are pointing at something structurally new: the normal academic attribution apparatus depends on influence being legible. If AI intermediaries can soak up ideas from private conversations, synthesize them, and produce outputs that are formally correct but intellectually unattributable, we don't have norms for that situation yet. This specific case may or may not involve misconduct. But the structural problem it reveals exists independently of OpenAI's behavior.
I think I'm seeing a pattern of illegal behavior here.
TL/DR: Mathematician opted out of training on 29-JUN and asked OpenAI whether they trained on his data and was told that it "did not happen" but it clearly did.
I've been suspecting over the last couple of years of the frontier companies using data for training anyway, regardless of training-use consent. "Using" the data doesn't have to mean they literally upload chat transcripts into pretraining datasets. My analogy has been money laundering -- if that can happen at massive scales, surely these companies can and will do the digital/data equivalent derivations/transformations. Even if one could have the access etc. to do so, how exactly would one prove that a given synthetic dataset that OAI/Anthropic uses is derived from particular user conversations that did not consent for the info to be used in training?
Consider, for instance that OpenAI's (consumer) terms say "If you do not want us to use your Content to train our models, you can opt out by following the instructions in this article ." but they also do say "We may use Content to provide, maintain, develop, and improve our Services". [1]
If you think that's quibbling, consider that OpenAI's business terms, in contrast, do state "OpenAI will not use Customer Content to develop or improve the Services, unless Customer explicitly agrees to such use.". [2]
[1] https://archive.is/EcwD8 [2] https://archive.is/yZdAF
And humanities have a word for this, exploitation, or appropriation, maybe it's time scientists and engineers revisited basic ethical notions. Skimming a dozen threads and nobody seems to have this vocabulary or willing to say it.
Well, OpenAI said "we didn't read the conversations", but they never discarded that the model was training with that data... so even worse.
Trust and OpenAI never go together in the same sentence. The answer is always no.
But China steals our AI!!!!!!
I think this is mathematicians coming to grips with the fact that AI is surpassing them
we will all have this moment soon enough, and it will change how we think about intelligence, identity and value
Or the companies hosting the frontier AIs are leeching the conversations.
Hopefully OpenAI can solve good problems where no one can make a claim that they stole their idea where the methodologies used and the techniques used are so out of the ordinary that the achievement is respected. Furthermore they should work on new novel solution on the old problems to lay these issues rest, thus by improving their models they can avoid issues of academics accusing them of using their work additionally the academics should also demonstrate their unpublished work is significant enough to have solved the problem . This is a bit subjective but it is also objective for the person with domain knowledge.
This is just mental illness at this point. I don't blame the mathematicians that have found a way to get attention from the mainstream press for once, but we should not fall for it here.
1. No one but OpenAI has produced a proof of NS so these accusations of plagiarism are pretty embarrassing. It reminds me of the line from the Social Network: "If they invented Facebook then why didn't they invent Facebook?". If these people proved NS before OpenAI where is their proof?
2. If they plagiarised Andreas Thom then why was his initial response to praise the proof and talk about how different it was from his own attempt? It's only now that it is clear that no one bothers checking these things that suddenly his story changes.