I don’t think it’s nearly as impressive as it is hyped to be:
They only ever evaluate on 100 prompts, and the error bars on their headline metrics are +-4-5.
The headline result (65%) only applies to Qwen3-0.6B chat; for GPT-4o, it’s 11%, and for Llama-2, 17%.
It’s fine-tuned on 400 ShareGPT prompts, and I suspect most of the gains are due to learning the ShareGPT prompt format. Indeed, if fine-tuned on a different dataset, it only achieves 11%.
The paper also has several misleading claims, like calling it “black-box” (needs the tokenizer) or “data-free” (only during pretraining, but see their own appendix).
Imagine if you could take the AI marketing slop, or political smear campaigns, texts, and Images, and see the misanthropic sludge that was used as a prompt to generate it.
I don’t think it’s nearly as impressive as it is hyped to be:
They only ever evaluate on 100 prompts, and the error bars on their headline metrics are +-4-5. The headline result (65%) only applies to Qwen3-0.6B chat; for GPT-4o, it’s 11%, and for Llama-2, 17%. It’s fine-tuned on 400 ShareGPT prompts, and I suspect most of the gains are due to learning the ShareGPT prompt format. Indeed, if fine-tuned on a different dataset, it only achieves 11%.
The paper also has several misleading claims, like calling it “black-box” (needs the tokenizer) or “data-free” (only during pretraining, but see their own appendix).
Imagine if you could take the AI marketing slop, or political smear campaigns, texts, and Images, and see the misanthropic sludge that was used as a prompt to generate it.
Or what if you could take something not slopped and get back the prompt that would most closely result in it
There is that. https://huggingface.co/spaces/pharmapsychotic/CLIP-Interroga...
Sadly we don’t really have that for T5XXL and other yuge prompt encoder based models
That's just a summary.