The goal of the AI act is not to determine if an "oh yeah!" comment was AI generated. The target is long papers that falsely claim human review and can have real significant consequences.
E.g. research paper, law makers, lawyers, state policies, notaries,...
These are much longer content and thus statistically they will disclose a better guess at AI generated content.
Asking another AI to paraphrase will not erase the mark (which they are unaware about) but rather cumulatively add their own mark and make it easier to detect.
The problem is not to use AI, but to endorse the responsibility of the content you (as a human) deliver and somehow make sure that fake-news, biased content or unverified output is detected as early as possible.
You'd think that the humans being paid to review these things are actually reviewing them. Journals, laywers etc. are expensive. With the addition of AI, it should be easier than ever to review things on their merits.
Maybe some part of it is that the deluge of slop is uncovering how poorly/sloppily these social institutions were working in the first place.
I feel like this is going to end up being like cookie laws. It sounds good, I don't know how any one benefits from it.
Currently I can recognize AI text because I read thousands of ai generated text. I know that 110% of yahoo finance news is generated. I don't want to read an AI generated personal blog, but if I do what's the problem really? Other than the companies distinguishing AI text for getting better training data, how do people benefit from watermarked text exactly?
This feature is born out of the same government that created the cookie laws, so it makes sense to me that it would follow similar politically-valuable but practically-questionable set of beliefs.
Really, Anthropic (and the other labs that follow) are just trying to satisfy the requirements of the law so they can continue to serve the EU. Wether it's actually effective is something else entirely.
I for one value the transparency of being able to tell human-written content apart, especially when moving into a time where we’re suddenly questioning an AI agent's motivations. If that is ingrained so deeply into these systems they cannot rip it out without us noticing, there’s at least one more safeguard. I know how ludicrous this sounds, but you can’t deny AI development is speeding up to scary levels.
Additionally, think of cases like paying a lawyer or an expert for an extensive report or opinion on something. Wouldn’t you want to know if that is actually their carefully assembled professional assessment rather than the output of an LLM prompt?
None of this is science or evidence based, it's political theater. The labs do this because the government has made it clear it will intervene if they don't say the line. I suspect most of the people at the labs think AI watermarking is retarded.
The people that make the laws and the people complaining about AI don't know or care about the reality of the situation nearly as much as they care about being re-elected and feeling good about their social posture.
AI watermarks feel like they're approaching the problem from the wrong side - no matter what it will be possible to remove the watermark (Via manual rewriting, local LLLMs, etc). Instead it seems like we need "proof of human creation". And the only way I see that being possible is hardware-attested proof of keypresses. Which obviously has huge privacy implications, but how else would you actually know that a piece of content was produced by a human pressing keys on a keyboard? The proof would also need to include timestamps for the keypresses, so tell if someone is just copying from another window.
The article mentions you can always simply use a smaller, local, un-watermarked LLM to rephrase the original watermarked text. Which is true, sure.
But if we're talking about deterministically taking some watermarked LLM output and having a function removeWatermark(text), it won't necessarily be "trivial" to remove, because the watermark function itself need not be public. Only the API that tests for the watermark need be public, right?
Anthropic's magic watermark could be, like the article mentions, something like "every 7th semicolon has a N% chance to be a comma where N is the sum of the last X characters mod Y, and every character in the bit range q1...q2 has a Z% chance to..." etc etc etc. And if Anthropic controls those variables, it would be very difficult to determine the rule, even with some pretty advanced analysis (I would assume). And keep in mind, that example rule I mentioned is pretty naive, too. I expect the actual rule would be way more advanced and not so straightforward as "swap every <charX> for a <charY>"
I don't know if it's possible to achieve a watermark that is undetectable, has a low false positive rate, and survives a wholesale rephrasing. Can you make a statistical measure that reliably survives 95+% of the words being different and the sentences reordered? Of course the more of the content you replace, the lower the quality, but in many cases you probably care more about the meaning of the text than the exact choice of words.
There exist methods to detect what kind of watermarking tool that someone is using, and most of the big tools have specific signatures that you can look for.
For Anthropic, it’s highly likely that the watermark is a SynthID type mark similar to the one that Sean is talking about (I actually ran the analysis here https://johnjwang.com/post/2026/08/12/how-claude-watermarkin...). When we get confirmation of whether all models actually are watermarked, I think we’ll be even more confident.
Of course it’s always possible that Anthropic has come up with a proprietary scheme, but I think it’s definitely harder to implement.
I think the game will be a cat and mouse game similar to LinkedIn and other websites trying to block scrapers: each iteration makes it harder for someone to figure out the watermarking scheme, but likely not impossible
It could be, but common, we all know that Anthropic's watermark is the using "load bearing", "genuine", and "seam" 1000x more in the same paragraph than any human in history.
I even see the Gemini diamond watermark on so many fake social media profile pictures. You could ask an LLM to find a github project that removes/inpaints those watermarks and be done with it. Or you could just use the API, which doesn't stamp the visible diamond on it (just the invisible synthID).
Most people are generally lazy. They upload text to LinkedIn full of "genuine", "honest" and "load-bearing".
Yes! How many people have been caught committing fraud because they used a modern word processor and fonts to forge old documents? How many leaks have occurred because people failed to redact documents correctly, despite there being easy to use tools for this very purpose? How many people neglected to strip sensitive EXIF information from images they share (before websites started doing it for them)? How many people flat out post evidence of their crimes on social media?
Yes, this watermark will be easy to strip. It is still valuable for the vast majority of times where people just don't.
Can it work on code itself? Obviously if you get AI to write the PR description that could be watermarked, but code isn't going to like random unicode, and the sythID approach feels like it would fall apart in the quite strict syntax of most code.
If a language supports unicode it will have a fairly permissive definition of whitespace, and it will be easy to generate permutations of the whitespace that meet the syntax requirements.
Of course I would want my code formatting tool to normalize that all to plain 0x20 spaces. But it would still be a helpful "brown M&M" test of did you even read CONTRIBUTING and run the code formatter before submitting this PR?
"What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around."
It would look like a lot of little signatures on little bits of text, and then larger signatures on a collection of those chunks once the larger chunk exists. It's not that hard. It's just a lot of signatures.
But this scheme could only ever prove that this bit of text was made by a given AI, and validate anything else ever included in the signature hasn't been tampered with. It's not hard to work up a scheme that proves (within reason) a text was generated no earlier than some date by incorporating some sort of information that could only have been known at that date so that could be validated. But this isn't even a step in the direction of proving that something was made by a human. And that's assuming the private keys stay private, which is its own tricky problem. If a private key ever leaks anything signed with it becomes invalidated.
> And that's assuming the private keys stay private, which is its own tricky problem. If a private key ever leaks anything signed with it becomes invalidated.
We don't appear to be talking about cryptographic signing here. (That would never work for the problem because everyone expects unsigned text anyway.) We're talking about:
> It’s basically a text steganography problem (concealing a secret code), made more difficult because the plaintext cannot be arbitrarily manipulated.
As in, trying to force ChatGPT output to contain intentionally crafted ChatGPT-specific LLMisms that a human is unlikely to imitate, even one who reads a lot of ChatGPT output.
> It would look like a lot of little signatures on little bits of text, and then larger signatures on a collection of those chunks once the larger chunk exists. It's not that hard. It's just a lot of signatures.
But how is this implemented? It's a few lines of code to implement a basic "identify and strip/replace any non printable ascii, unicode ..." or whatever.
A screenshot/OCR will also do this.
SO at the end of the day you're left with some dumb rules like "you used `load-bearing` more than once per 500 words, that's AI!"
To quote the original article's context more fully:
"The output of chat tools (and most of the output of AI agents) is not containerized text, but plain old regular text, and so can’t be signed. What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around."
I'm reacting to the idea that "plain old regular text" can't be "signed" because they aren't "files". I'm observing that you can sign a stream of text, and probably other metadata, no problem. To my reading this really is about signing and not stegonographic watermarking, so we're in a context where for some reason the users in question want to carry the certificate of generation by AI and so the fact that this is trivially strippable isn't the issue at hand.
I read it this way because it seems to me clear that it isn't any particularly harder to do the stenographic stuff on a stream than a file (per zahlman's comment), so it only makes sense to be talking about this if we are actually talking about signing.
It's done with a lot more subtlety and embedded directly into the content, no Unicode shenanigans. The basic idea is just to break token generations where the probability is nearly tied in favor of the side that matches the secret key. With a long enough text block you can be statistically certain if the generation was using the key. From another comment: https://johnjwang.com/post/2026/08/12/how-claude-watermarkin...
> AI companies already store all prompts and responses for future training.
They claim to only do this when you agree to this in your personal settings. Though Google does say they will train on it, unless you disable history and only use ephemeral chats. Anthropic has a setting for it and claims not to train by default.
Also it would be vulnerable to attacks and privacy problems. You could search for substrings about some suspected information, like "John Smith's medical records show advanced cancer" etc. Of course you'd have to guess the phrasing but still.
Wouldn't this become pretty difficult to do at scale over time? Is there a way to compare the similarity in a database of responses without doing a search over every entry and comparing them? Because that would probably become pretty slow if literally every LLM output is saved and has to be scanned.
Society can overcome this problem by changing the way we think about text. Raw plain text should be banned, all text is cryptographically signed by the editor, gui element, or tool that created it.
For SynthID and similar solutions, there is much I don't understand ...
Here's what I grasp: The AI system scores each token and then selects tokens based on those scores. If we encode something in the token selection routine ('in order choose the 1st, 3rd, 1st, 5th, 2nd, then 1st highest scored tokens'), we can identify AI-generated text by comparing sample text (ST) to the expected text (ET) for that prompt.
1) How do we score the tokens for the ET without the original prompt? Even a Markov-like process needs to start somewhere.
2) To recreate ET don't we need to maintain, until the end of time, the AI state - entire model and code - at the time of ST output?
3) Doesn't #2 require maintaining all states for all AIs? Often you won't know when and from which AI system the ST might have been generated. What happens when an AI vendor goes out of business?
4) To recreate ET, don't we effectively have to rerun the prompt? Won't rerunning it for every verification increase most costs of AI output by an order of magnitude? Most of what AI vendors do would be ST validation.
1. So we can't score the first paragraph (or similar-sized block), and not short texts? Not deal-breaker, but a limitation.
2. Doesn't the score vary by each AI system state - its model, programming, harness, etc.? Claude's output today doesn't match Gemini's, nor Claude from 2 years ago.
Yeah but it's like saying "Masterlocks will always be easy to pop off with a hammer". Of course, but by doing so you are actively engaging in fraud, which then puts the onus on you and whoever you are attempting to deceive.
Key phrase. And I'm not saying fraud in the legal liability sense. If you're not trying to hide the fact that something was LLM generated, then you have no reason to remove it. If you are trying to hide it, then there's probably a reason, i.e. you would face consequences for doing so, therefore it is fraud.
Or, you simply edit the text the LLM generated ruining the hidden message. That's not even remotely fraud. There are legitimate reasons to edit text. There are fewer legit reasons to bash off someone else's lock with a hammer.
The goal of the AI act is not to determine if an "oh yeah!" comment was AI generated. The target is long papers that falsely claim human review and can have real significant consequences.
E.g. research paper, law makers, lawyers, state policies, notaries,...
These are much longer content and thus statistically they will disclose a better guess at AI generated content.
Asking another AI to paraphrase will not erase the mark (which they are unaware about) but rather cumulatively add their own mark and make it easier to detect.
The problem is not to use AI, but to endorse the responsibility of the content you (as a human) deliver and somehow make sure that fake-news, biased content or unverified output is detected as early as possible.
You'd think that the humans being paid to review these things are actually reviewing them. Journals, laywers etc. are expensive. With the addition of AI, it should be easier than ever to review things on their merits.
Maybe some part of it is that the deluge of slop is uncovering how poorly/sloppily these social institutions were working in the first place.
I feel like this is going to end up being like cookie laws. It sounds good, I don't know how any one benefits from it.
Currently I can recognize AI text because I read thousands of ai generated text. I know that 110% of yahoo finance news is generated. I don't want to read an AI generated personal blog, but if I do what's the problem really? Other than the companies distinguishing AI text for getting better training data, how do people benefit from watermarked text exactly?
This feature is born out of the same government that created the cookie laws, so it makes sense to me that it would follow similar politically-valuable but practically-questionable set of beliefs.
Really, Anthropic (and the other labs that follow) are just trying to satisfy the requirements of the law so they can continue to serve the EU. Wether it's actually effective is something else entirely.
I for one value the transparency of being able to tell human-written content apart, especially when moving into a time where we’re suddenly questioning an AI agent's motivations. If that is ingrained so deeply into these systems they cannot rip it out without us noticing, there’s at least one more safeguard. I know how ludicrous this sounds, but you can’t deny AI development is speeding up to scary levels.
Additionally, think of cases like paying a lawyer or an expert for an extensive report or opinion on something. Wouldn’t you want to know if that is actually their carefully assembled professional assessment rather than the output of an LLM prompt?
None of this is science or evidence based, it's political theater. The labs do this because the government has made it clear it will intervene if they don't say the line. I suspect most of the people at the labs think AI watermarking is retarded.
The people that make the laws and the people complaining about AI don't know or care about the reality of the situation nearly as much as they care about being re-elected and feeling good about their social posture.
AI watermarks feel like they're approaching the problem from the wrong side - no matter what it will be possible to remove the watermark (Via manual rewriting, local LLLMs, etc). Instead it seems like we need "proof of human creation". And the only way I see that being possible is hardware-attested proof of keypresses. Which obviously has huge privacy implications, but how else would you actually know that a piece of content was produced by a human pressing keys on a keyboard? The proof would also need to include timestamps for the keypresses, so tell if someone is just copying from another window.
The article mentions you can always simply use a smaller, local, un-watermarked LLM to rephrase the original watermarked text. Which is true, sure.
But if we're talking about deterministically taking some watermarked LLM output and having a function removeWatermark(text), it won't necessarily be "trivial" to remove, because the watermark function itself need not be public. Only the API that tests for the watermark need be public, right?
Anthropic's magic watermark could be, like the article mentions, something like "every 7th semicolon has a N% chance to be a comma where N is the sum of the last X characters mod Y, and every character in the bit range q1...q2 has a Z% chance to..." etc etc etc. And if Anthropic controls those variables, it would be very difficult to determine the rule, even with some pretty advanced analysis (I would assume). And keep in mind, that example rule I mentioned is pretty naive, too. I expect the actual rule would be way more advanced and not so straightforward as "swap every <charX> for a <charY>"
I don't know if it's possible to achieve a watermark that is undetectable, has a low false positive rate, and survives a wholesale rephrasing. Can you make a statistical measure that reliably survives 95+% of the words being different and the sentences reordered? Of course the more of the content you replace, the lower the quality, but in many cases you probably care more about the meaning of the text than the exact choice of words.
There exist methods to detect what kind of watermarking tool that someone is using, and most of the big tools have specific signatures that you can look for.
For Anthropic, it’s highly likely that the watermark is a SynthID type mark similar to the one that Sean is talking about (I actually ran the analysis here https://johnjwang.com/post/2026/08/12/how-claude-watermarkin...). When we get confirmation of whether all models actually are watermarked, I think we’ll be even more confident.
Of course it’s always possible that Anthropic has come up with a proprietary scheme, but I think it’s definitely harder to implement.
I think the game will be a cat and mouse game similar to LinkedIn and other websites trying to block scrapers: each iteration makes it harder for someone to figure out the watermarking scheme, but likely not impossible
If you can submit the text to determine whether it's watermarked, you just have to progressively alter the content more and more until it passes.
It could be, but common, we all know that Anthropic's watermark is the using "load bearing", "genuine", and "seam" 1000x more in the same paragraph than any human in history.
Yeah, but it's better than nothing.
People underestimate the value of rules that only take malice and a little knowledge to break.
I even see the Gemini diamond watermark on so many fake social media profile pictures. You could ask an LLM to find a github project that removes/inpaints those watermarks and be done with it. Or you could just use the API, which doesn't stamp the visible diamond on it (just the invisible synthID).
Most people are generally lazy. They upload text to LinkedIn full of "genuine", "honest" and "load-bearing".
Yes! How many people have been caught committing fraud because they used a modern word processor and fonts to forge old documents? How many leaks have occurred because people failed to redact documents correctly, despite there being easy to use tools for this very purpose? How many people neglected to strip sensitive EXIF information from images they share (before websites started doing it for them)? How many people flat out post evidence of their crimes on social media?
Yes, this watermark will be easy to strip. It is still valuable for the vast majority of times where people just don't.
Exactly. If this works on pull requests, for instance, it'd be really useful for projects trying to do a first-pass filter to close slop spam.
Can it work on code itself? Obviously if you get AI to write the PR description that could be watermarked, but code isn't going to like random unicode, and the sythID approach feels like it would fall apart in the quite strict syntax of most code.
If a language supports unicode it will have a fairly permissive definition of whitespace, and it will be easy to generate permutations of the whitespace that meet the syntax requirements.
Of course I would want my code formatting tool to normalize that all to plain 0x20 spaces. But it would still be a helpful "brown M&M" test of did you even read CONTRIBUTING and run the code formatter before submitting this PR?
"What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around."
It would look like a lot of little signatures on little bits of text, and then larger signatures on a collection of those chunks once the larger chunk exists. It's not that hard. It's just a lot of signatures.
But this scheme could only ever prove that this bit of text was made by a given AI, and validate anything else ever included in the signature hasn't been tampered with. It's not hard to work up a scheme that proves (within reason) a text was generated no earlier than some date by incorporating some sort of information that could only have been known at that date so that could be validated. But this isn't even a step in the direction of proving that something was made by a human. And that's assuming the private keys stay private, which is its own tricky problem. If a private key ever leaks anything signed with it becomes invalidated.
> And that's assuming the private keys stay private, which is its own tricky problem. If a private key ever leaks anything signed with it becomes invalidated.
We don't appear to be talking about cryptographic signing here. (That would never work for the problem because everyone expects unsigned text anyway.) We're talking about:
> It’s basically a text steganography problem (concealing a secret code), made more difficult because the plaintext cannot be arbitrarily manipulated.
As in, trying to force ChatGPT output to contain intentionally crafted ChatGPT-specific LLMisms that a human is unlikely to imitate, even one who reads a lot of ChatGPT output.
> It would look like a lot of little signatures on little bits of text, and then larger signatures on a collection of those chunks once the larger chunk exists. It's not that hard. It's just a lot of signatures.
But how is this implemented? It's a few lines of code to implement a basic "identify and strip/replace any non printable ascii, unicode ..." or whatever. A screenshot/OCR will also do this.
SO at the end of the day you're left with some dumb rules like "you used `load-bearing` more than once per 500 words, that's AI!"
To quote the original article's context more fully:
"The output of chat tools (and most of the output of AI agents) is not containerized text, but plain old regular text, and so can’t be signed. What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around."
I'm reacting to the idea that "plain old regular text" can't be "signed" because they aren't "files". I'm observing that you can sign a stream of text, and probably other metadata, no problem. To my reading this really is about signing and not stegonographic watermarking, so we're in a context where for some reason the users in question want to carry the certificate of generation by AI and so the fact that this is trivially strippable isn't the issue at hand.
I read it this way because it seems to me clear that it isn't any particularly harder to do the stenographic stuff on a stream than a file (per zahlman's comment), so it only makes sense to be talking about this if we are actually talking about signing.
It's done with a lot more subtlety and embedded directly into the content, no Unicode shenanigans. The basic idea is just to break token generations where the probability is nearly tied in favor of the side that matches the secret key. With a long enough text block you can be statistically certain if the generation was using the key. From another comment: https://johnjwang.com/post/2026/08/12/how-claude-watermarkin...
If everyone is using ai (per reported revenues) and nobody like the outcome (banned here, dismissed there), then what is the future of AI?
Anthropic the other day announced future models "launched in the EU" will include watermarking https://support.claude.com/en/articles/16266773-how-claude-m...
I was assuming it was something like SynthID rather than just sneaky invisible unicode but it's hard to tell from the description.
I done a C2PA implementation for raw text, if that can ease someone’s life: https://github.com/dualeai/c2patxt
here‘s what i don‘t get about this whole discussion. AI companies already store all prompts and responses for future training.
just make an API that returns the string distance between a previously generated paragraph and the query?
that would sidestep this whole problem class.
regulators could even specify how that has to work.
what am i missing?
> AI companies already store all prompts and responses for future training.
They claim to only do this when you agree to this in your personal settings. Though Google does say they will train on it, unless you disable history and only use ephemeral chats. Anthropic has a setting for it and claims not to train by default.
Also it would be vulnerable to attacks and privacy problems. You could search for substrings about some suspected information, like "John Smith's medical records show advanced cancer" etc. Of course you'd have to guess the phrasing but still.
Wouldn't this become pretty difficult to do at scale over time? Is there a way to compare the similarity in a database of responses without doing a search over every entry and comparing them? Because that would probably become pretty slow if literally every LLM output is saved and has to be scanned.
1. AI company buys and trains on an author’s book when it gets published, it’s now part of the training data.
2. Attacker asks the LLM for the opening sentences of the book, it goes into the generated responses database.
3. Later, a malicious user shows that the first few sentences of the authors book are identical to a previously generated response.
> AI companies already store all prompts and responses for future training.
They store some prompts and responses, not all, that's what you're missing.
Local models?
Local models enable:
- watermark-free generation
- the stripping of watermarking from the output of SAAS models
Any discussion of watermarking is dead in the water in a world where we are permitted to have these things. I fear for the future.
Chill my dude. This is just a sane default which will catch normies copy pasting stuff from claude and chatgpt. It's good enough.
Good. Tracking and surveillance have no place in the modern world.
Society can overcome this problem by changing the way we think about text. Raw plain text should be banned, all text is cryptographically signed by the editor, gui element, or tool that created it.
You can ask an AI agent to type into Microsoft Word via computer use.
For SynthID and similar solutions, there is much I don't understand ...
Here's what I grasp: The AI system scores each token and then selects tokens based on those scores. If we encode something in the token selection routine ('in order choose the 1st, 3rd, 1st, 5th, 2nd, then 1st highest scored tokens'), we can identify AI-generated text by comparing sample text (ST) to the expected text (ET) for that prompt.
1) How do we score the tokens for the ET without the original prompt? Even a Markov-like process needs to start somewhere.
2) To recreate ET don't we need to maintain, until the end of time, the AI state - entire model and code - at the time of ST output?
3) Doesn't #2 require maintaining all states for all AIs? Often you won't know when and from which AI system the ST might have been generated. What happens when an AI vendor goes out of business?
4) To recreate ET, don't we effectively have to rerun the prompt? Won't rerunning it for every verification increase most costs of AI output by an order of magnitude? Most of what AI vendors do would be ST validation.
1. score by the past paragraph (this is also how the verifier checks without needing a GPU, it takes the previous length of text)
2. same answer
3. no, it just needs the previous text, private key, and the matrix math (CPU is fine)
4. no, see above.
Thanks ...
1. So we can't score the first paragraph (or similar-sized block), and not short texts? Not deal-breaker, but a limitation.
2. Doesn't the score vary by each AI system state - its model, programming, harness, etc.? Claude's output today doesn't match Gemini's, nor Claude from 2 years ago.
Yeah but it's like saying "Masterlocks will always be easy to pop off with a hammer". Of course, but by doing so you are actively engaging in fraud, which then puts the onus on you and whoever you are attempting to deceive.
Except this isn't like saying that at all. This isn't fraud, because it's legal in almost every circumstance.
>almost every circumstance
Key phrase. And I'm not saying fraud in the legal liability sense. If you're not trying to hide the fact that something was LLM generated, then you have no reason to remove it. If you are trying to hide it, then there's probably a reason, i.e. you would face consequences for doing so, therefore it is fraud.
Or, you simply edit the text the LLM generated ruining the hidden message. That's not even remotely fraud. There are legitimate reasons to edit text. There are fewer legit reasons to bash off someone else's lock with a hammer.