Please report success/failure after each test. Asking me to read and compare 30 writing samples to get any feedback at all means I won't finish. Telling me immediately when I got one wrong lets me recognize patterns and improve my guesses.
Utterly imperceptible, even when studied under the microscope in a way that LLM text very rarely is in practice.
It will be interesting to see whose concerns are assuaged (perhaps they genuinely though mistakenly believed it would degrade quality), and whose concerns are heightened (perhaps their real objection is that their AI-generated text will become detectable).
If anyone notices degraded quality, that would imply they could break the crypto behind the watermark.
For an analogy, distinguishing AES ciphertext from random bits without the key would be counted as breaking AES (the more precise statement of this is called AEAD).
I'm not sure the watermark has been demonstrated to have that kind of property. Even if you use a CSRNG as the 'key' once you're feeding it through the token selection process it's going to risk re-introducing certain correlations.
I've read that watermarking should in theory be impossible to detect except by the entity that watermarked it. Which is sensible, and I mostly understand at a high level.
But what I don't know and don't understand is what happens if you watermark watermarked text. Does it test positive for both watermarks? Only the second? Indeterminate?
Or maybe I'm misunderstanding. Can you tell that it's watermarked, but only the entity who put the watermark in place can test if it's theirs? My confusion about watermarking multiple times still stands, though.
Regardless of what happens when you watermark multiple times, no matter the outcome, it weakens the watermark. Which, depending on the threat model, kind of makes it moot. I can't imagine a serious situation where a watermark can be weakened in any way and still be useful. Even "this came from an LLM" isn't a valid signal if you can just watermark ANY text through purely mechanical means.
It's also not clear to me how this will affect mainstream LLMs. If all output text is watermarked, there MUST be an escape hatch. Otherwise, JSON schemas will break (or provide holes where unwatermarked text can be exfiltrated through MCP), "return this text exactly with no changes" will be impossible, and writing diffs will break.
If you are refering in terms of the code of practice part of the EU AI Act
> I've read that watermarking should in theory be impossible to detect except by the entity that watermarked it
This is a carveout exception, for watermarking. In the spirit of those terms it should be machine identifiable.
In my opinion they should have thought better about this, paricularly for text, because in its current forms it is easy to lead next to a new "tamper-proof" requirement, which in practice is DRM. And we do not need more DRM.
For images, music there is metadata already where such information can be stored. And if end users are found using unlabeled AI their accounts could be ban from these platforms. Not something the social platforms might want, but it's a saner approach than trying to reinvent the secret printer dots on all generated media.
i think practical question is whether the detector survives ordinary transformations of the text. if for say i paraphrase a watermarked answer with another model, do we expect the original signal to disappear and the second model's signal to replace it?
There are multiple ways to paraphrase and the paraphrasing model is going to use its own random number generator whenever it thinks there's more than one possible choice. (Not really binary; the RNG will have more or less effect.)
So it will definitely be watermarked by the paraphrasing model. But the question is whether the original signal survives at all. There might be a weak signal that's detectable with enough text?
No. Anthropic's example is completing the sentence "Isaac Newton's most famous work was called Principia ..." has only one correct answer, so nothing to watermark.
The watermarking is an inextricable part of the token generation, it just using a known pseudo random sequence for the sampling. It’s not a transform that can be applied later.
This obviously stops working as soon as you don't have the entire context. To reliably detect a subset of the LLM's output you need to do something more sophisticated but also more invasive.
This is like giving you three outputs from md5sum and asking you to guess for which one the input ended in a "q". There's no way to tell unless you break the RNG.
I think the point is probably to help convince people that the watermarking doesn’t perceptibly impact quality, which is a concern some people have (whether well founded or not).
Wow I actually got a 7/10. It was hard to tell at first but there are signs that tipped me off to which one probably had a higher score out of the multiple choice.
My understanding is that watermarking in prose is basically a bias when sampling tokens. For a system that knows the average probability for each possible token in the LLM vocabulary it is possbile to quantify said bias given enough text.
For a human that doesn't reason in tokens and therefore doesn't know anything about their probability distribution, it should be impossible to tell. Relying on fancy words/constructs within sentences should not give you any signal as well, since you don't know if the the prompt included instructions for that.
I was never going to do very well on this. My ADHD was itching after the third one. I suspect it would have been sooner but I had a bit of extra focus from the suprise that it selected an answer for the first question when I tried to scroll.
To avoid that on the following questions I just held my finger on my phone to avoid a click. That eventually selected some text, and I instinctively tapped to deselect. That triggered another random pick, then I just tapped through to the end because I was fed up.
My sense of the concern here is that watermarking may somehow deprive someone or something of value regardless of whether or not they can tell, so I briefly pondered trying to rank these from best to worst and see if any set of those votes meaningfully deviated from ~average.
That said, I read the first triple and found all three tortured enough that I can't be bothered with the rest.
Google's SynthID page says they can watermark text, but it also says that it can only detect the watermark on "image, video or audio". Does that mean that the text watermarks can't actually be used as watermarks?
I disliked the fact that the experiment only covered prose, which my eyes glossed over and made me actually do random entries to pass on and see the results. I'd love to see it on a more accurate output distribution like commented code
It was only at question #8 that I realized there was a third option, and while I’d love to say that accounts for how I got a 1/10, after reviewing the third options I doubt it would’ve made a difference.
Please report success/failure after each test. Asking me to read and compare 30 writing samples to get any feedback at all means I won't finish. Telling me immediately when I got one wrong lets me recognize patterns and improve my guesses.
I did five, then gave up and just pressed A until I reached the end. I got 3/5 right.
Exactly the same here. 4/5.
Yea, I did two and then harrumphed in annoyance that I was expected to do all 10.
We must do something about this immediately! Immediately! Immediately!
https://www.youtube.com/watch?v=jLO7VrRij_M
> Telling me immediately when I got one wrong lets me recognize patterns and improve my guesses.
Wouldn't this make it a worse measurement?
Yes. Did one - saw that I wouldn’t get feedback until I have completed all 10 (if at all) and noped out.
I believe this is to test the theory that people can detect watermarked text.
Teaching you how to identify watermarked text while the experiment is running would ruin the data.
Utterly imperceptible, even when studied under the microscope in a way that LLM text very rarely is in practice.
It will be interesting to see whose concerns are assuaged (perhaps they genuinely though mistakenly believed it would degrade quality), and whose concerns are heightened (perhaps their real objection is that their AI-generated text will become detectable).
If anyone notices degraded quality, that would imply they could break the crypto behind the watermark.
For an analogy, distinguishing AES ciphertext from random bits without the key would be counted as breaking AES (the more precise statement of this is called AEAD).
I'm not sure the watermark has been demonstrated to have that kind of property. Even if you use a CSRNG as the 'key' once you're feeding it through the token selection process it's going to risk re-introducing certain correlations.
I've read that watermarking should in theory be impossible to detect except by the entity that watermarked it. Which is sensible, and I mostly understand at a high level.
But what I don't know and don't understand is what happens if you watermark watermarked text. Does it test positive for both watermarks? Only the second? Indeterminate?
Or maybe I'm misunderstanding. Can you tell that it's watermarked, but only the entity who put the watermark in place can test if it's theirs? My confusion about watermarking multiple times still stands, though.
Regardless of what happens when you watermark multiple times, no matter the outcome, it weakens the watermark. Which, depending on the threat model, kind of makes it moot. I can't imagine a serious situation where a watermark can be weakened in any way and still be useful. Even "this came from an LLM" isn't a valid signal if you can just watermark ANY text through purely mechanical means.
It's also not clear to me how this will affect mainstream LLMs. If all output text is watermarked, there MUST be an escape hatch. Otherwise, JSON schemas will break (or provide holes where unwatermarked text can be exfiltrated through MCP), "return this text exactly with no changes" will be impossible, and writing diffs will break.
I feel like I must be missing something.
If you are refering in terms of the code of practice part of the EU AI Act
> I've read that watermarking should in theory be impossible to detect except by the entity that watermarked it
This is a carveout exception, for watermarking. In the spirit of those terms it should be machine identifiable.
In my opinion they should have thought better about this, paricularly for text, because in its current forms it is easy to lead next to a new "tamper-proof" requirement, which in practice is DRM. And we do not need more DRM.
For images, music there is metadata already where such information can be stored. And if end users are found using unlabeled AI their accounts could be ban from these platforms. Not something the social platforms might want, but it's a saner approach than trying to reinvent the secret printer dots on all generated media.
Watermarking only applies where the AI has a free choice (https://www.anthropic.com/news/claude-text-watermark).
i think practical question is whether the detector survives ordinary transformations of the text. if for say i paraphrase a watermarked answer with another model, do we expect the original signal to disappear and the second model's signal to replace it?
There are multiple ways to paraphrase and the paraphrasing model is going to use its own random number generator whenever it thinks there's more than one possible choice. (Not really binary; the RNG will have more or less effect.)
So it will definitely be watermarked by the paraphrasing model. But the question is whether the original signal survives at all. There might be a weak signal that's detectable with enough text?
If its paraphrased to any significant degree I'd expect the original to not survive. The second model's watermark would of course be there regardless.
Does it still apply with zero entropy?
No. Anthropic's example is completing the sentence "Isaac Newton's most famous work was called Principia ..." has only one correct answer, so nothing to watermark.
The watermarking is an inextricable part of the token generation, it just using a known pseudo random sequence for the sampling. It’s not a transform that can be applied later.
This obviously stops working as soon as you don't have the entire context. To reliably detect a subset of the LLM's output you need to do something more sophisticated but also more invasive.
This is like giving you three outputs from md5sum and asking you to guess for which one the input ended in a "q". There's no way to tell unless you break the RNG.
Yeah, it's a pointless exercise. I hope the author is just trolling given that he is knowledgeable in the field.
> Here are three 64-character hex strings. Two are random. One is HMAC-SHA256(secret_key, "anthropic"). You don't have the key. Which one is the HMAC?
I think the point is probably to help convince people that the watermarking doesn’t perceptibly impact quality, which is a concern some people have (whether well founded or not).
Wow I actually got a 7/10. It was hard to tell at first but there are signs that tipped me off to which one probably had a higher score out of the multiple choice.
You likely just got lucky.
If SynthID is a google technology, then this is likely just us training their ai again, captcha all over again.
Could do with some context on how watermarking works. Objectively speaking it should be impossible to tell.
My understanding is that watermarking in prose is basically a bias when sampling tokens. For a system that knows the average probability for each possible token in the LLM vocabulary it is possbile to quantify said bias given enough text.
For a human that doesn't reason in tokens and therefore doesn't know anything about their probability distribution, it should be impossible to tell. Relying on fancy words/constructs within sentences should not give you any signal as well, since you don't know if the the prompt included instructions for that.
I was never going to do very well on this. My ADHD was itching after the third one. I suspect it would have been sooner but I had a bit of extra focus from the suprise that it selected an answer for the first question when I tried to scroll.
To avoid that on the following questions I just held my finger on my phone to avoid a click. That eventually selected some text, and I instinctively tapped to deselect. That triggered another random pick, then I just tapped through to the end because I was fed up.
My sense of the concern here is that watermarking may somehow deprive someone or something of value regardless of whether or not they can tell, so I briefly pondered trying to rank these from best to worst and see if any set of those votes meaningfully deviated from ~average.
That said, I read the first triple and found all three tortured enough that I can't be bothered with the rest.
Call me persuaded, I guess.
Interesting, I did very badly, 3/10!
Same, doing worse than random chance seems like an interesting signal though, but I'm not sure what it's a signal of.
Only one third of the options at each stage are watermarked, so 3/10 seems well within random chance.
Google's SynthID page says they can watermark text, but it also says that it can only detect the watermark on "image, video or audio". Does that mean that the text watermarks can't actually be used as watermarks?
It feels all of them are terribly written. I don't know why.
Because it's AI slop? Not that surprising.
This is stupid --- way too long and wordy. Make your test worth taking.
I disliked the fact that the experiment only covered prose, which my eyes glossed over and made me actually do random entries to pass on and see the results. I'd love to see it on a more accurate output distribution like commented code
I don't think anyone is, or plans to be, watermarking AI-generated code as opposed to text.
https://www.anthropic.com/news/claude-text-watermark
> code—which in very many cases has to be exact—has generally less watermarking than some other forms of text.
Generally less watermarking. Not no watermarking.
What do you mean? The watermarking applies to all text outputs, including code.
It’s just much less effective since code is low entropy.
Why not? It would help with a lot of potential legal issues.
It takes a lot of patience to read this much slop voluntarily.
Thought there were only 2 options!
Me too lol.
It was only at question #8 that I realized there was a third option, and while I’d love to say that accounts for how I got a 1/10, after reviewing the third options I doubt it would’ve made a difference.