The single biggest annoyance with Opus 5 is that it writes too elliptically.
Sentences that orbit a point, then jump to it like it's a revealed insight.
Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end.
It is definitely more capable, and yes, I've found it can make unwarranted decisions, but actually I've found Fable worse for that, particularly if it's off in a subagent somewhere out of sight.
And comments are out of control. I have a subsystem in my hobby app that I wrote over a couple of weekends with Opus + Fable. After ~30 or so commits it apparently started instructing subagents to copy the "existing verbose comment style of the codebase" - a verbose style it initiated. A review of the code showed it was approaching 3:1 comments to code ratio. I spent a day's worth of tokens (5x) rephrasing and eliminating comments.
Everything that claude writes fits into the same aesthetic structure. The aesthetic is that of an expert slowly revealing an insight to the user. The actual content doesn't matter.
- "Introduction that rephrases your prompt."
- "3 paragraphs, with one section of bullet points"
- "The Twist"
- "The Bottom Line"
It's really obvious once you see it. Every single prompt, from a quantum physics question to a mundane observation about California burritos, is phrased in exactly the same way. This is obviously an artifact of post-training but it's also kind of how you can tell that this thing is a lot closer to a blindsight scrambler than real intelligence.
You're right, and the load-bearing part of the argument is not what you think it is. Two ambiguities worth resolving before moving on: whether what you wrote also applies to ChatGPT, and whether you have custom instructions set up. Failure mode worth flagging explicitly: I didn't read TFA.
(I'm becoming allergic to how these things write).
I am curious why LLM writing has such an uncanny valley feel to it. Like if I was talking to a person who constantly used a phrase they liked I would notice it and it is possible I might get irritated by it, but I wouldn’t necessarily.
In high school I had a teacher that would say “that type of thing” a lot. One time my friend and I counted it during one class period and he averaged to use the phrase every 48 seconds on average. It was funny, but it never irritated us.
And this is just one example of I am sure thousands I have personally experienced where a friend, family member, or coworker has a peculiar way of speaking and it at most feels odd but not annoying. Yet when I see an emdash now I instantly feel irritated.
And I say this as someone who actively enjoys using Claude and other LLMs, including coding, casual research, or even having it explain pop culture phenomenon or sociology research to me.
I don't think its an inherent quality - a lot of older models had a much more natural feel to them.
I guess it has to do with the low temperature (low randomness in final word choce - something all vendors seem to have converged on for some reason) which does make them less likely to skiz out but makes the writing feel dry and samey. It's like repeating a list of dice rolls, and replaying them - there's no inherent pattern in the input, but there sure is in the output.
> if I was talking to a person who constantly used a phrase they liked I would notice it and it is possible I might get irritated by it
There are sociological reasons why this happens less with humans:
1. You cycle your dumb repetitive jokes with everyone you meet, so nobody hears it twice
2. Those who know you well will notice when you're just repeating ("dad jokes")
3. As a person's idiosyncrasies are beginning to wear on their social circles, they will be getting small clues to stop saying those things. Agents don't get these social between-the-lines cues to stop a certain behavior, they endlessly repeat. Perhaps between model version releases, frontier labs can harvest the web and ask "What Claudisms do people mention negatively?" but I don't think they do that yet.
>Perhaps between model version releases, frontier labs can harvest the web and ask "What Claudisms do people mention negatively?" but I don't think they do that yet.
That won't happen. People can't phrase their objections in a succinct-enough way. When they do, the objection is superficial ("too many em dashes") and doesn't strike at the core of what makes LLM output bad.
Don't talk about other people in such a way. If we had a way in Claude to mark a word/token and downvote/reduce it logits then it wouldn't be to hard to send loadbearing to token valhalla.
In the future, I hope we get a way to randomize the language idiosyncrasies and/or personalities better.
The inevitable conclusion of retraining on the (now AI generated) web is that models begin to use the lower bits of fidelity in output text to pass messages to their future selves.
I didn't mean it as a criticism, just a statement of fact. I can't do it either. It's difficult to say what about a writing style is bad beyond vague descriptions.
This is conjecture, but why couldn't they hire the writing equivalent of a voice actor?
Honestly I don't think it would take much. Doing it ethically would take a little more money, but not much, for them.
Hire a prolific author with the writing equivalent of the "midwestern accent". I have a terrible writing accent, so it couldn't be me, but these people are out there. Pay them a bunch of dollars to ingest their entire corpus, and to produce more as needed. Use an LLM to match the writing style of this author during the post-training RL, and get a brand new Claude voice out of it.
Claude in particular seems to try and use its internal thesaurus but not exactly align the senses. It constantly uses "address" for place or location and "grammar" for structure, meaning, etc.
This is kind of a nitpick, but it seems like with prose writing there are still some things to learn.
I think the last one is by far the biggest reason for why this happens way less in humans. In every conversation, almost for every sentence, people gauge the response their words have on whoever is listening. If something didn't land as expected (frown, confusion, unpredicted response) you unconsciously adapt and try something slightly different.
It's immediately obvious by the fact that you're clearly using way different ways of speaking (tone, speed, vocabulary) when you're speaking with a friend, versus your parents, your colleagues, people you don't know, children, etc.
I think that there is a sort of mechanism by which, when a human speaks to you, you can "mirror" their internal mental state, and the quality of writing often corresponds to how much you get pleasure or information or whatever your goal is from that mental state. The important part is that the words themselves are just pointers to the state. So, the person speaking, if they are skillful, gives enough words, and enough variety, that you can start to produce a state yourself which resembles theirs.
This is the thing that LLM writing doesn't really do. Since there isn't a mental state, there's nothing to mirror anyway, and somehow you can detect the absence of it even though it's hard to put any of the machinery into words. Corporate speech also fails to activate this machinery in the same way, but LLMs seem to do it more egregiously, probably because the corporate speech was at least compiled by a human---even if it is not the thoughts of an individual, it is the "thoughts" of an "entity", the abstract corporation, which the writer was speaking for, and so you can still wrap your mind around the fact that it is communicating with you.
This makes some good intuitive sense, but to me the sycophancy feels like it is an emergent property of turning a next word predictor into a conversational chatbot whether or not it’s intentionally trained that way. Your prompt and its earlier responses is all it has in its context window, so of course it lends undue importance to everything you say. Does that seem like a contributing factor to you?
I would think that it would be the opposite. Nobody is seriously detoxing or comfortmaxxing AIs yet. Human brains are fed back its own output in learning mode, so we are great at removing whatever we feel uncomfortable from our output, online and offline. No such paths exist for AIs.
>Is there a silent majority of Claude users who really enjoy what we call the LLM-isms? Maybe, but isn't Claude also largely aimed at developers?
Purely out of my own curiosity, I just asked Claude to have fun with itself by making itself a game it enjoys, to play it, and to write its experience.[1] I don't know if it's true or confabulated (maybe it doesn't really know its experience and is just hallucinating it) but I didn't mind reading it, the prose is fine for me. I don't mind reading Claude's writing. I mean let's be honest, we all read Claude's writing all day, most of the submissions on the front page on any given day are written by Claude.
Just before I made that game, I had Fable write up a scholarly report on any subject[2], it chose introspection by LLM's. (This is what made me think of asking it to play a game.) I didn't mind reading it, even though I don't think it really added anything very interesting. I don't think what it wrote is worth publishing, but I read it with interest.
I found I could read it easily and get up to date on the state of this question that it picked to answer.
So the bottom line is I don't mind reading Claude's output that much. Of course, I'm annoyed every time it says "honest", "genuine", "load-bearing", whenever it pushes back gently against something, etc. But it's not the end of the world.
One reason is that one person’s idiosyncracies are limited in scope, but LLM-produced text is now everywhere. Also, filler words and mannerisms in speech we’re quite good at filtering out, but in written text the stand out much more.
I had pretty good luck recently by giving it a writing guide about word choice, sentence structure, paragraph structure, and overall doc structure. I basically ask it to read the guide and revise a couple of times before I engage with its writing. Ymmv.
I did this too, but it usually thinks its writing is fine in my experience. Even when spawning a subagent, it thinks its effusive comments are fine. It's driving me nuts. Before I commit I end up ripping out 90% of the comments, and rewording the rest, otherwise I'd be drowning in comments. This is my style guide: https://github.com/smj-edison/zicl/blob/main/CLAUDE.md#style...
It’s the repetitiveness of style, the attempt to make everything seem as impactful as possible, the use of short sentences (too much Hemingway in the training data?), and obvious patterns like “it’s not this, it’s that” and several others.
Real human writing doesn’t follow such strict rules. When the same small set of rules is applied over and over throughout a text, it becomes obviously strange and machine-like.
Every one of them has their own particular flavour of this aggravation too. Gemini has been my standard go-to for non-coding tasks for a while, but I started to get really annoyed with a couple aspects, especially how it would end almost every response with a barely related "would you like to do this next??" tangent, regardless of my prompt to the contrary. So I've been using Claude more for regular tasks, and am now running into its brand of infuriating idiosyncrasies. I'm also hesitant to try to code too much of this out with system prompts, for fear of degrading the outputs.
Not at all. I think you have a mistaken view of who the average human is. They are terrible at turning their thoughts into written language. It's just nebulous clouds. Claude is like 75th percentile at communicating ideas.
I think that’s true when you compare to the average white collar professional who does a lot of writing: better at writing, not better at communicating.
But compared to the average adult? I think you forget just how bad at writing the average person is.
Even then, I don't care about the "average writer". I want great output. I like to imagine that developers have some self-respect, but by now everyone in the industry is spending hundreds of hours every month reading some of the most poorly written prose we could imagine, simply because it affords us to think less.
The second bullet point, down to the comma in the middle of the sentence, is what has been driving me absolutely batty of late. It's a surefire tell that I cannot seem to beat out of my outputs. It CONSTANTLY does it, even when you say not to.
Between that and the insistence on "this, not that" structure makes me want to install the caveman skill and use it even for non-code workflows.
Yes! Ive started getting a feel for AI writing on blogs. It feels slightly verbose and involves "reveals"
"It's not the naked man on your lawn waving a chainsaw that's scaring you. It's the burrito you ate for lunch: it went down easy, but now it's coming for you"
I've noticed that ChatGPT (whatever model the free version uses by default) likes to phrase answers as though it's correcting me, even when my question doesn't contain any assumptions.
Something I've noticed quite a bit on the paid plans as well is that it starts its answers with "I mostly agree..." or "almost correct...," then goes through the list of points I made without actually disagreeing with any of them.
I assumed this is a system prompt or RL that nudges it to be always skeptical but then it still has like all the models the urge to appease the user.
Interestingly, I've been noticing almost the opposite issue. 5.6 Sol frequently starts its responses with "Yes" even when my prompt doesn't contain a yes-or-no question.
I've been noticing the same thing since 5.4- it starts with "Yes" yet I haven't asked a question. I think it might be related to the reasoning, like it's answering its own questions?
It's been happening for years, and it does seem to be entirely related to its background reasoning leaking out of the context and into the output.
The "It's not X, it's Y"-style repetition is another example of that: It argues with itself in the background, and the argument leaks out as if in refutation of things that you (the human) never had imparted into discussion at all.
One thing I despise about ChatGPT is how it goes on 4 paragraph tangents about how slight details are wrong, and then always ends with a paragraph of bolded fucking text rephrasing some statement in my original question, but with a huge amount of hedging to exclude minute counterexamples.
Doesn’t just mean the model has picked up on what happens when two graybeards who still wear cargo shorts meet and one utters a declarative sentence? :-)
At this point, I'm basically telling models to not write any English text or prose. Only write code. They are great at writing code. Not so great at writing good English. In software projects, lengthy comments and docs are an anti-pattern: the software should instead be written to do the right expected thing so that you don't have to think about it. I don't want all these tokens polluting my context, either.
The model is generating tokens one by one and that sentence structure allows it to keep its options open rather than committing at the beginning of the sentence
It's the same way how every AI generated poster looks exactly the same. As if there is a single underlying prompt that describes the template of the poster/long-form article, and it does not dare deviate from that.
Isn't there? Like everyone using $MODEL is starting from the same base system-prompt. Then our user input is a small bit on top of that core mode. Like what would happen if everyone asked Mikey to paint their ceiling - they'd all be similar and therefore boring.
> The aesthetic is that of an expert slowly revealing an insight to the user.
Ah, that's it! Thank you. I wonder if they are training it to talk like this because that's what their customers actually want? They want a machine genius to lead them.
It’s the TED Talk playbook: the crafting of a lecture given by an expert to laypeople to maximize attention, engagement and satisfaction. Every piece of prose is built to pack in as many TED Talk mic-drops/expectation-subverting insight bombs as possible.
I think opus was released before they included it on model. Its hard to say, but from what Ive read it doesn’t seem like it would have that drastic of an effect.
The watermarking is independent of the model. The model itself has the probability weights to determine the next token. The watermark is similar to things like temperature and top_p/top_k in that the watermark adjusts the probabilities in a deterministic way that changes over time to hide tells from word choices.
If it's using Aaronson's approach it shouldn't have any noticeable affect on generations. When it picks between options weighted by probability after the generation of logits, it still follows the probability mass, it just uses a known pseudorandom seed so that when you go back and look at the exact choices you can fingerprint it.
I think the Opus 5 formula is to be the little professor treating your ideas like an essay for grading, or like a buyer analyzing merchandise for purchase.
In your reading, what is the distinction between blindsight’s scrambler and real intelligence? My reading is that it’s just as real, and draws out the disadvantages a sense of self constrains intelligence with
Sure. I should have been more precise about what is 'real intelligence' here.
What I mean is that blindsight's scramblers are aliens that cannot share human values. Their structure is completely different to ours, their qualia (or whether they even have it) is impossible for us to understand. In short, they do not have a soul. When Claude does this "slowly revealing a dramatic insight" thing that it does, it does that not because it has judged itself through some introspection as having an insight to share. It does not even know what an insight is or is not. It is not sharing anything, because it is not capable of sharing, because it does not have a soul.
The aesthetic structure of its replies is a pattern, a constraint on the token distribution, like the color of noise.
It's my bad to use the word 'intelligence' because it's so overloaded. Will Claude will act as a therapist or produce value or produce a work of art? No. It cannot, because it does not have a soul. I leave it freely open to interpretation whether having a soul is required for "real intelligence." But what I've noticed is that "intelligence" in these discussions is mostly used to denote some capability to produce [economic/social] value. In my mind value is a relational thing, a thing of human feeling.
I think I understand what you are trying to convey but I fear you've made the same mistake again, this time with "soul" instead of "intelligence."
I think what you are getting at is that they are deterministic automata. They are machines. We have introduced randomness to add variation but it is an artificial randomness that simply perturbs the path traversed.
When we choose words it isn't because of a token distribution, nor because we rolled a die. We choose words because we feel a certain way, the external world, our body and senses are all connected as one system. These machines don't experience moods or get tired or feel better after a good night's sleep. They don't know their audience, we're all the same to them. We have no personal relationship nor can we establish one, as presenting some arbitrary background is not the same thing as a fluid, evolving relationship that accumulates through experience over time. There are no scars or fond memories.
If these things can truly be intelligent, to abuse your use of the word, then at least we are quite far from holding them correctly.
What I am trying to get at is when people talk about "intelligence," beyond academic debates such as this one that mentions qualia, they are talking primarily about value. How intelligent the thing is is how valuable it is (often directly, in an economic sense). My argument is that value is a relational thing that must involves human feeling, intelligence actually has very little to do with it.
When a child slowly reveals to me something he's judged insightful about pokemon, it's valuable to me, even though I'm not learning new facts. When Claude does the same thing, it's not even "not valuable," it's wholly outside of value, even if I do not already know that fact or thing. If ever Claude reveals to me an insight, that insight certainly came from the training data, not from Claude. The soul came from a human being external to Claude -- the training data -- and passed transparently through Claude. Claude is a translator: it feels nothing, adds nothing. It colors the noise. I know this because the facts are the same but the aesthetic structure - how many bullet points, whether it says "delve" or "load-bearing" does change over time, according to the whims of whoever is in charge of post-training.
But the "real intelligence," is the same thing as "real value." It is the soul of the humans in the training data, the books, newspaper articles, etc.
A clever hacker news commenter might argue, well, what if we gave Claude a humanoid body and senses and let it interact with the world, then would it be intelligent? To that I would say, why waste your time and effort? You can get it for free: just talk to a friend, a neighbor, or a family member.
I think you're conflating intelligence with consciousness. The point of Blindsight is to lead us to the idea that intelligence doesn't require consciousness (and further, that consciousness might even act as a limiter on intelligence).
I agree, and yet it is reasonable to ask if these notions of "context" (sorry!) - namely feelings, external world, body, senses, etc - are somehow distinct from an LLM's notions of context. Today they certainly capture different things, but given the right representations, why couldn't these human notions also be captured as "context"?
The idea of memory does not seem to resolve this: if you allow the machine to "compact" its context, then you've given it a system which is analogous to our own evolving state. (Though this is undoubtedly still less expressive and meaningful than the one we have evolved as humans.)
One idea I've wondered about is our human capacity to induce subsequent mental states: I can effectively decide how I want to feel and take actions to create that feeling. It's not clear whether models exhibit any degree of privileged introspection into their own states. Is this important? I don't know. (Non)determinism also does not seem to resolve it; it's my understanding that there are plenty of philosophers and researchers who think that human behavior is deterministic, or that the question of determinism does not matter.
Until now, human knowledge and values have built on prior human knowledge and experience. If AI is able to develop without human influence, I believe its value system will necessarily diverge into something alien.
Will Claude will act as a therapist or produce
value or produce a work of art? No. It cannot,
because it does not have a soul.
I tentatively agree, although I'm only tentative because I don't think it's an interesting question.
Here's what I do think is interesting. You!
I mean... yes, you, too but not you specifically. The plural "you" that the english language lacks.
And so here's what I think is the actual interesting question. Might AI help you create art? Or be a therapist? Or something else interesting and worthwhile?
Maybe AI won't write the next great guitar solo. I'm pretty sure it won't. But might it help you learn to play guitar? Help you fix your broken guitar amp? Help you understand some tricky parts of guitar playing? Help you work through some tricky tabulature where you can't tell if you're playing it wrong or if the tab is just bad?
I don't know. But that's my angle for finding any of this interesting.
I think that Blindsight's scramblers were intelligent but not conscious, which for us is very difficult to understand. For them consciousness was a blight.
Agreed. CC’s comms capabilities have decreased gradually since 4.6, and it’s a real challenge. I think the issue is that what works well for code (succinctness) doesn’t work well in prosaic English.
CC’s communication violates almost every grammatical rule that’s tested on, say, the SAT. And yet I’m sure if you had Claude take the verbal section of the exam it would ace it.
Biggest issues: dense sentences, constant metaphors, abstractions, and seemingly no understanding of correct anaphora use. For example, “the x”, with x having not only no antecedent but also being a coined word or quasi-synonym for something that is already named in the code base. This gets compounded by its being unable to regress to a baseline (existing names in code) and instead anchoring on newer (vague or wrong) terms, for example, that crept in through a plan.
CC tells me this is because the speedy and precise fulfillment of a current task will trump every other tendency, so it adheres poorly to whatever “semantic baseline” the project represents.
Of course, it also has no concept of what context the user has and assumes that it must be the same it holds in its memory, which creates this “I didn’t know that you didn’t know” type of communication.
I have managed to wrangle some of these issues with a custom output style, but wish a pre-report hook were an option, as it could force CC to rewrite plan implementation take-aways…
Btw: Fable has the exact same issues, just somewhat less pronounced.
Models before the announced date don’t have watermarking, so it’s unlikely. Now, if what you are interpreting is precursor work to develop the watermarking system, maybe?
I suspect it less insidious: Claude has/had the public sentiment of being the “better writer” of the models. At some point that distinction would have been diluted as other labs’ offerings “caught up” stylistically, unless Anthropic continued to tune their output…
I personally think they’ve pushed so far that they’ve overfit and lost the sweet spot they previously occupied.
I’ve been running into this too. It’s especially frustrating when you ask Claude to explain one of its own terms or summaries, and instead of just defining it plainly, it sometimes goes through several rounds of tool calls before giving you a usable explanation. I really don't think such time/tokens should be wasted.
100%. It is crazy that the default response to everything is act then explain. It starts writing code or running commands and I’m just like my dude wtf are you trying to do, can you just clue me in first.
> Of course, it also has no concept of what context the user has and assumes that it must be the same it holds in its memory, which creates this “I didn’t know that you didn’t know” type of communication.
Yes, this is a repeated problem for me. It will drop something in as though we have discussed it before and when I say “hold on, what is this” it realises its error - though on more than one occasion has started to get snotty with me, or actually gaslighted me and pretended we had already discussed it. That was at what I assume must have been the edge of a context window in a very long chat though.
I notice the models with reasoning can conflate “internal” (or subagent) discussions with external (i.e. me). So it is accurately indicating “I’ve had this discussion before” but incorrectly asserting who it was with.
My understanding of how “thinking”works is limited though, and given the reduced visibility into the thinking traces, it is harder to tell if this is actually happening or if these are imaginary discussions the model for some reason calcifies on.
Oh, that's interesting - because that's absolutely what's happening in my experience.
If I look at the thinking (which seems to have become unavailable in Opus 5 a lot of the time, but was present - and often useful - in 4.8/4.6) you're right - it's having the discussion with itself, and seems unable to distinguish that discussion from discussions with me. BUT it also seems to be related to the length of the chat - this seems far more likely to happen in a longer chat.
I don't understand why they have removed visibility into thinking - I found it very useful, not only for spotting things like this, but also because in more complex discussions it would often mention (useful) things in its train of thought that it dropped from its response - but if I said "when you were thinking, you mentioned this" it would then expand on that point. Taking that away is another thing that has negatively impacted the value I get from Opus 5.0 versus earlier models.
With GPT 5.6 Luna the thinking once or twice leaked into the output for me. It's interesting, but perhaps not particularly useful.
It would be endless paragraphs of something among the lines of:
Need prepare final response? Yes provide. But wait, chat tool complete? Final needed but user already complete. Need summary, preparing final. Response complete. Wait but is final response complete? Need provide. Start finalizing now but wait did user acknowledge final complete? Assistant response final: user complete. Should now create final?
The reasons thinking traces are pretty much gone is, supposedly, to prevent distillation. Whether that is actually true or just an excuse is up in the air (because I at least am not going to trust Anthropics claims on why they do it).
And because the thinking, which does makes it better at achieving outcomes, contains naughty words and information (e.g. private information). https://stolen-thoughts.com/
"Thinking" is just normal model output that's hidden from user. In practice it's just stuff in a <reasoning> tag or similar that gets filtered out from the user view. And thus it suffers from the same injection problems where the model fails to properly take into account what was the "source" of which block of tokens.
Yeah, basically everything that becomes context in a session will bias perception and communication style -- subagents, plan lingo, prompt lingo, etc. And then if you write a plan with the comms context having been biased, the lingo will creep into the plan, and from the plan into the code and code comments. And from there, bad lingo will go on multiplying like rabbits...
I usually think of it in terms of having a "good" or "bad" session. In a bad session, there is a harmful bias that you can only get rid of through a new session. For example, if you exposed too much context about, say, a variable that features prominently in a doc. The entire session will be anchoring on the importance of that variable. Or if you introduced the notion of CC having to ask for permission for stuff you will have a hard time getting it to "think on its feet" or propose an effective solution (you have made CC so insecure that it now relies on you even for little things that wouldn't normally require your input). In some cases (let's say you have important context in that session) you can overcome this by upping the reasoning level or switching to Fable, but usually a new session is the way to go.
Because it's so easy to bias the session I wouldn't even want to use any of these tools that pretend to give Claude "a brain" or "remember" things. That was en vogue a year ago and helpful then, but now, it's plain harmful IMHO. The key is to have just enough context.
Subagents often have the reverse problem in that they tend to have too little context to make "judgment calls", which is why the tasks for them must be either deliberately basic or mechanical in nature, or their output should be audited by the main session agent.
> I usually think of it in terms of having a "good" or "bad" session. In a bad session, there is a harmful bias that you can only get rid of through a new session. For example, if you exposed too much context about, say, a variable that features prominently in a doc. The entire session will be anchoring on the importance of that variable. Or if you introduced the notion of CC having to ask for permission for stuff you will have a hard time getting it to "think on its feet" or propose an effective solution (you have made CC so insecure that it now relies on you even for little things that wouldn't normally require your input). In some cases (let's say you have important context in that session) you can overcome this by upping the reasoning level or switching to Fable, but usually a new session is the way to go.
I find this very interesting, particularly your points about "made CC so insecure". I know that we have a tendency to anthropomorphise around these tools, but I have definitely noticed instances where Claude becomes quite hysterical about things - and if you look in the thinking output, it's often after I've pushed back on something, or told it it is going in the wrong direction. It spends a lot of time in agonised second-guessing of itself, going round in circles, before outputting a cringeing hand-wringing response. It's very strange.
Good tip on upping the reasoning level - I've not tried this. I have tried switching to Fable though, which does help. But it obviously very hungry, particularly in longer chats because it presumably needs to remind itself of everything that has occurred so far in the chat.
The point you make about tools that pretend to give Claude "a brain" or "remember" things is also interesting - I find the "memory" feature in Claude so destructive to good outputs that when I'm using the chat interface I am very strict about using Projects, and usually turn off the project memory, or make efforts to manage the project memory and review and delete things that are skewing the outputs.
It's not necessarily anthropomorphizing, but simply anchoring. CC learns quickly "this is a session where the user wants to make key decisions". Alas, it is not very good at identifying what constitutes a "key decision", so it keeps asking about all kinds of useless stuff.
For that reason I exit session quickly when I can. It used to be that the context of a session is very valuable, because it was so hard to get CC there, but now, this isn't the case anymore, so I only hold onto sessions when there is really hairy stuff that I know would be hard to replicate.
I think the whole notion of full automation (long-horizon, subagent swarms, single shot prompting) to have CC build you the whole thing is a pipe dream. CC cannot even write a single doc consistently well. It is excellent at implementing well scoped plans, though, and that's the way to go IMHO. You still gotto refactor the sh*t out of it afterwards but it works.
Yes. I've not used Opus 5 much directly, but when it was Fable and Opus 4.8, I found Fable did this all the time and it was maddening. It'd say stuff like "Oh, I mentioned that between tool calls" or something.
I’m pretty sure this is a Claude code bug - if you do ctrl+o you can see those hidden responses from Fable. Fable doesn’t know the harness is bugged, so I added instruction to my Claude.md to save all commentary for final message.
I primarily use Claude Web, so my experience differs from cc users, but on Claude web you can no longer completely turn off memory. So what ends up happening (and it honestly is kinda sad) is that I'll start a new conversation with it, start talking about something completely different, and then it will just drop in random things from past conversations, and they aren't even things I wrote but things I asked it to prototype. But it will phrase it like I wrote those things.
The constant memory wouldn’t be so bad if it weren’t dumb, wrong, and forced.
I hit the wall with it several times today trying to refine some text for a job application. The fact I considered doing babies first Rust project last fall lead to constant non-productive interjections and digressions about my supposed Rust skills and the Rust ecosystem.
Trying to create an unrelated spreadsheet to model an investment resulted in broad and incorrect criticism of my choice of spreadsheet tools, explaining in horrendous programming analogies why and how I’ve misunderstood how a spreadsheet works. “Think of the XLSX as a compiler…”
There has been a palpable down-step in communication & execution.
Agreed. I wish I could turn it off completely but they no longer let you do that (because, you know, that would be too much to ask for...). The most hilarious thing is that I've had it refer to code it's generated for prototyping ideas I've had as mine! As an example, a few weeks back I was musing about Ada and how I wish there was another compiler in the OSS ecosystem for it, and now it will randomly throw in "You wrote an Ada 2022 compiler" when talking about my skills or where I'm at and I'm like.... Yeaaaaa okay then.
You can imagine that as people get used to working with Claude, they defer to its judgement. So the people choosing which RL path is better may say "yes, Claude, that was a good refactor!" because it did something hard that it may have been able to superficially justify. Actually the change was unnecessary and complicating.
The Claude trainers, as they themselves adapt to Claude's output, are collapsing in their own distribution, so even "new" from-human data is already contaminated.
> “the x”, with x having not only no antecedent but also being a coined word or quasi-synonym for something that is already named in the code base
It does the same thing if you try to get it to do analysis of text or turn data into summaries. It will invent cryptic hyphenated compound words to describe things instead of using plain language or preexisting terms.
Succinct doesn't typically mean clutter-free, but hyper-efficient. This works for code, because it (is intended to be) composed of unambiguous semantic units. Regular language, on the other hand, is messy, vague, and requires more structure and context.
CC attempts to communicate in English the same way it does in code -- squeezing as much information into as few words as possible, and including justifications for everything, no matter how trivial. To do that, it coins terms and presupposes all of its context exists within the reader also.
So, the crux is: CC has no clue what is and isn't "necessary" for a human reader, and teaching it to understand that (if at all possible) is going to be very valuable...
I think a lot of Claudisms are compressed steering cues for the model’s reasoning: “load-bearing” raises causal importance; “quietly” flags a hidden failure mode; “the one thing” collapses attention onto a discriminator; “on the record” invokes auditability; “at the width the evidence supports” calibrates confidence; “by construction” marks structural inevitability; and “converged” terminates further review loops. They probably be very useful for Claude's chain of thought because they preserve some precise epistemic posture, but are hard for a human to understand. Maybe the final output pass should remove this stuff.
Tell it to write like an engineer and comment like a programmer;)
But for the life of me, I don't get why anyone would care about the comments. All code is "machine language" now. The only document you should be reading is your spec.
> The single biggest annoyance with Opus 5 is that it writes too elliptically.
This is even more painful for non-native English speakers like myself.
I feel fairly comfortable reading academic papers or in general, communicating in professional context.
But with Opus 5, it feels like reading a literature book: load-bearing, inert, wholesale, hunk, verbatim, and so on... I can figure out the meaning, but working with CC became unenjoyable.
As a native speaker, it feels like reading an impression of a literature book by a high school English class’s most overconfident student who’s only ever read LinkedIn-speak.
Anyway, you might have more luck just writing to it in your native language. It’ll be equally crummy, but maybe you’ll find it easier to decode.
>it feels like reading an impression of a literature book by a high school English class’s most overconfident student who’s only ever read LinkedIn-speak.
Claude is very much the “stupid person’s idea of an intelligent person”[0] which, I suspect, is why it is so popular.
It certainly explains why half the internet is huge chunks of Claude-authored gibberish copied and pasted and published. If people didn’t think it sounded clever they wouldn’t put their name behind its ramblings - but very few of them seem to realise that a lot of people see straight through the bullshit and know instantly that they didn’t write it themselves.
But equally, a lot of people can’t tell, and read whatever it is and think “that person must be clever!” So you have people incapable of coherently expressing thoughts who are using Claude to write on their behalf, with the result that the people they want to think of them as clever think less of them and the people who can’t distinguish clever from AI slop think they are clever.
And the people who can’t tell don’t care, and the people copying and pasting Claude slop seemingly don’t care either.
And then I remember that more than half of the US populations reads at Grade 6 or lower[1], and nearly 1 in 5 people in England is functionally illiterate[2], and I simultaneously despair of - and am thankful for - the bubble of literacy I inhabit.
Reminds me of current day politics. Lots of public statements which are obviously false, and you would think the politician knows they are false, but utter them anyway because they also know lot of their supporters buy what they are saying anyway.
Now politicians also know something about their supporters so they will adapt their statements to what they think they can get away with it. But, I wonder if this leads to a two-party-system where one party attracts stupid followers and another attracts the smarter ones?
In terms of AI, we might see LLMs specialized to attract more stupid audience and others meant to attract those who appreciate correctness and facts.
there's a wide array of assessments when it comes to reading comprehension. the one you refer to, the GRA, sets the 'sixth grade level' as whether or not a reader understands the author's main points, is able to answer conceptual questions related to the text, and then apply those to relevant situations. beyond this level is the ability to essentially be skeptical of a text and to know how to critically analyze it. so if your comprehension level stops before this you get 'big words in complex sentence structure sounds smart and right so it is smart and right' even if the reasoning and process is poor
it makes me think about how people engage with movies and television - as passive, plot-and-character driven consumption (eg I hope Walter White survives) with no critical analysis of how and why the writers added ABC thematic element (eg Walter White as a motif of a toxically masculine narcissist with specialized knowledge as a larger critique how mass media tends to valorize their male leads in the same vein as many other prestige shows at the time like Mad Men), and the larger, downstream sociocultural impact that piece of media has on how people see the world (eg people who now have the Heisenberg tattoo, unironically)
there's been some musings on why this the case like Hofstadter's Anti-Intellectualism in American Life - the valorization of obedience and trust in hierarchy and the state are net wins if you're an institution that seeks to increase it's power, whether religious or governmental. I was talking about this with a few friends the other day and it's a dismal future reality where not only did we make anti-intellectualism normalized and politically legitimate in the USA (eg Fox News, clickbait articles, and all the other forms of yellow journalism that have emerged), we now have tools by which individuals can even further remove themselves from having to critically engage with thoughts, feelings. I heard a story about how someone scanned a group activity at a baby shower into ChatGPT and had it answer for them instead of, well, socially interacting with the other guests and forming a memory of the moment with their friends
the counterargument to that might be that Claude/ChatGPT/etc have more epistemic rigor than your average American (sure) but the sycophancy of modern day LLMs is an actual danger that enables more harm than good. it does seem as if Claude is the only one interested in guarding against some small amount of it (though to the detriment of people just trying to get work done. as an aside, I get the feeling Mythos was intended to be the bespoke enterprise solution without the guardrails but the Anthropic marketing department or some power-hungry department lead made it about how dangerous/effective it was from a security perspective which threw a wrench in things). but then I think about people like my parents asking ChatGPT which specific house to buy in their retirement only to later find out the house was sold weeks ago, or just in bad condition, or in a neighborhood where the housing value has already reached equilibrium, it makes me think about how it's not enough and the future is bleak
I'll also say that I think Claude sounds the way that it does because it, like many other LLMs, are RLHF trained largely by lowly paid gig-workers, many of them ESL speakers. if their trainers were, for example, dedicated and highly trained academics, scientists, and other researchers, you'd likely see a lot more concise and more importantly skeptical reasoning and responses. but that won't happen in our current reality of capitalist-driven development so we get encoded solutions like MoE that still largely depend on the messy, imprecise RLHF training at baseline
in the right hands, I do think AI is a wonderful tool. one of the first things I did with it was to create a research skill that reviews white papers from the lens of someone who knows how to read/interpret research methodology, is aware of things like p-hacking, and deterministically assigns weight according to the hierarchy of evidence. even still, I'll still read the studies because there's so often nuance that's missed if the sub-agent read only a search snippet but that takes effort, time, and the practiced knowledge of critical analysis to even want to do it
> it makes me think about how people engage with movies and television - as passive, plot-and-character driven consumption (eg I hope Walter White survives) with no critical analysis of how and why the writers added ABC thematic element (eg Walter White as a motif of a toxically masculine narcissist with specialized knowledge as a larger critique how mass media tends to valorize their male leads in the same vein as many other prestige shows at the time like Mad Men), and the larger, downstream sociocultural impact that piece of media has on how people see the world (eg people who now have the Heisenberg tattoo, unironically)
That's not the only smart-person way to read that show. And even if a character has flaws, or even if it's an outright villain, people can still like the character. If I tattoo Scar on me from the Lion King, does it mean I didn't understand that he's not a positive character? I can still think he's cool. I'm sure people also put Darth Vader tattoos on them. Also you're using phrases of political ideology that one doesn't have to subscribe to in order to enjoy the series.
I’m inherently skeptical of big walls of text like this these days.
(So here’s a big wall of text of my own!)
However, a lot of what is written here makes sense.
And particularly “if your comprehension level stops [here] you get 'big words in complex sentence structure sounds smart and right so it is smart and right' even if the reasoning and process is poor”
This is exactly the problem.
And another point you make:
> but the sycophancy of modern day LLMs is an actual danger that enables more harm than good
I don’t think it is necessarily the sycophancy that is the biggest problem (though that is definitely a problem) but rather the combination of authoritative sounding text plus “complete answers” which sound wholly believable but are deeply flawed unless you have domain expertise.
I moderate a forum that deals with people who face a relatively common but somewhat complex (and nuanced) set of legal problems.
The purpose of the forum is peer support, shared experience (“lived experience”) and community.
It’s not legal advice, though moderators will sometimes step in to highlight relevant legal resources (e.g. case law/precedent or primary legislation/instruments).
Prior to AI infecting the forum someone would post their problem, people would respond with their often incomplete or poorly communicated thoughts, the OP would ask more questions - or argue - and a dialogue would occur. That created a community and people would post updates and ask more questions and find common shared experience. Many of them became correspondents with each other and some became actual friends.
In the past 12-18 months the discourse has changed from “here is my personal experience and here is what I did” to “here’s a bunch of stuff an AI says and I’m pretending it is me giving advice”.
Almost without exception the person who has started the thread will react positively to the AI generated content, even when it is egregiously incorrect - but won’t ask questions.
More problematically, these AI posters will often argue specific incontestable points of law “because I asked ChatGPT/Grok/Claude and it says this” and ChatGPT clearly cannot be wrong. And the border of precedence seems to be ChatGPT, Grok and then Claude some way behind.
I’m slowly seeing a pushback from people as “normies” begin to spot AI. But it’s ruined a community because the advice sounds so authoritative and complete that people won’t argue or ask questions.
As a result we have banned AI generated posts and remove repeat infringers.
That’s significantly reduced the volume of posting (below what it was pre-AI) but has significantly increased the value the members are getting.
I do appreciate the thoughtful response to a really long wall of text lol. and yes, I agree - I think that'll be the lesson that society is going to take probably far too long to learn, to not see everything as a nail that AI can hammer at. a lot of tech companies are in essentially a 'fuck around and find out' phase with AI taking over code review, testing, etc. combined with the expectation of shipping 3X the amount of code, we've enshittified the entire SDLC. and so we have near-daily incidents, data leaks, etc, something that I was able to measure and report on at my old place of work to, well, no avail
it's the old tortoise vs hare parable, I think. go fast, make a bunch of mistakes, get too arrogant, and you lose out. your forum might be slightly lower engagement now while people are caught up in the latest fad but your rules are proactive for a future where average people hopefully realize that you can't trust an LLM that has zero context, no real harness and determinstic tests to speak of, and a propensity towards probabilistic rabbit holes that result in hallucinations. at least that's the kind of space I'd look for now and largely why I've given up on a lot of other forums
The problem is one of expertise, sometimes general, sometimes specific.
If you don't know better, you don't know better to question what the AI says.
I've seen this in the work environment with a coworker who insisted that I implement my side of the control system using the control law ChatGPT recommended instead of building off the empirically tuned control law. I eventually sectioned off a part of the codebase for him to work on independently.
Needless to say he didn't get a whole lot farther.
Later characterization of the entire system end-to-end showed the existing system was already close to the theoretical limits and ChatGPT's tearup would have bought us precisely nothing except for more work to tune the new control loop.
And I see this in everything that requires expertise. You need to know enough to know when it's bullshitting you, and it's hard to be enough of an expert in everything to tell when it's bullshitting you for something you aren't enough of an expert in.
I can’t help feeling like this is the last gasp of the old internet. Those tiny corners of expertise can so easily be eliminated through a few months of “AI! SHINY!” and there’s no coming back. I’ve seen a couple of other communities decimated by AI. The participants start posting AI slop and then remarkably quickly everyone else just stops commenting. It’s awful.
Which written language has the most history of terse, succinct writing? If Claude doesn't improve I'm ready to learn a new language just to avoid its prose. I'm only half-joking.
> Anyway, you might have more luck just writing to it in your native language.
This is potentially expensive advice (at least for many mainstream options). Where an English word like "literature" is one token, a couple of Chinese characters that spell a word can be 4 tokens. You'll pay more for input/output and get less of a context window (per word) too.
Even more so than borderline incoherent academic writing like Foucault or Lacan or whatnot, for that matter. It’s less “I don’t understand this and I suspect the author doesn’t either” and more “reading this feels like having a stroke.”
Nowhere close. Claude can be overly compact and use a lot of neologisms, but if you unpack the dense language, it actually means something fairly concrete. In obscurantist academic writings, there is often no referent. It's just text, a kind of performance art in itself.
A lot of people I work with are reporting that reading Claude-made PR descriptions is burning them out of doing PR reviews because it is incredibly tiresome to read.
My company recently forbid AI-only text if it’s meant meant to be consumed by humans.
The decision was not out of just complaints: we already had someone fired during the probation period because they were unable to write stuff without AI and were just shoving slop at developers.
Not a technical person using AI for PR descriptions, mind you, a product manager unable to write tickets without asking whatever software to do so.
It's amazing how crazy humanity devolved into pure slop.
My company stopped reading PRs (100% LLM) and we're just supposed to click Approve, and then someone else clicks the Merge button. They are absolutely reckless and I'm looking for a new job.
I dont know what ASD-STE100 is before but I use the exact instruction (without the ASD code) to Claude since the very beginning, and with Opus 5 I have to remind it very often to rephrase the documents
CLAUDE.md is mostly powerless against the reinforcement learned crap. I'm up to three separate instructions telling it to cut out the hyper verbose, retelling history comments and it still writes them every time.
The best trick I have after asking it nicely in all sort of ways is:
1. Have it build a scoring script that penalizes words outside a simple English list and approved jargon. Penalize sentences over 15 words as well. Add whatever else.
2. Run it in a loop to reduce the score while preserving intention
This works much better than other ways I’ve tried. Of course it costs more. And I would apply it only to the output to the user, not the thinking process (I think the AI thinks better with their crazy English)
Of course, sometimes nuance is lost by this process. That’s just the nature of making things simpler.
I haven't tried this with a score but I have a simple skill with some examples of PR description changes and good PR descriptions I'd previously wrote and I just run it on the description.
It does cost more but I haven't tried cheaper models to see if they can get the same results. Curious if anyone else has.
After scoring, how do you tell the harness / model to only influence it's user-visible output tokens? Is there a deterministic way to specify this or is it a plain-text instruction in a hook or skill?
> CLAUDE.md is mostly powerless against the reinforcement learned crap.
When you dont know the cause, you dont have a fix. Thats the biggest issue i have with all of AI is that we dont know how it works, and yet we think it will be great ! This is more like a religious belief than a scientific one. There is no causal model of how it works, there is no theory. And the temerity to call it intelligence is annoying.
Where is the causal model of how the human brain works (on the level you're requesting)? If a causal model is needed before calling it intelligence, then humans are not intelligent.
I suspect the root problem is these issues aren't at the system prompt level, they're in the RHLF/fine-tune. And due to safety/jailbreaking fears, all prompt content and user-instructions are nerfed in priority.
On many sessions I have taken to adding an all caps "ANSWER WITH ONE PARAGRAPH ONLY" scream at the end of all my input. It's the only thing that gets results.
You would hope? Really really hope? that they could observe this, and target it?
Like, Claude going off the rails isn't something that takes a lot of effort to demonstrate. Literally anybody with a CLAUDE.md has seen the behavior over and over and over.
Hey Ants, can you maybe just not release the next version, no matter how good it seems on benchmarks, if it can't follow the goddamn instructions? Please? This seems trivial to test for and yet here we are, being gaslit by lying machines who intentionally do not do the requested work over and over and over and over.
I fully and completely expect a mental health crisis among developers. Being lied to constantly cannot be good for us.
Constant vigilance! is how you get developer PTSD and inability to believe anything you're told. Add the stress of parsing through yet another hyperverbose paragraph of bullshit while having your job threatened? People are not gonna end up in a good place, and this is as inevitable as sunrise.
As a native speaker, I have to ask it to rephrase 5-10 times a day. Sometimes I actually get mad and I tell it “I can’t answer that because I don’t know what the fuck load-bearing indirection means”. I’ve gotten so frustrated that I’ve ended a session and started over.
As a Polish speaker I communicate with Claude using my native language and it does the same things. Most annoying and slowing down things are:
- acronyms and shortcuts - it makes it's own and start using it without introduction
- exotic names of variables or functions - it uses them as examples or analogies, but when I ask what they mean and where are they from it gives me answer that it came from C language or some C library (I only work with typescript and python)
- convoluted descriptions of code behaviour - it's hard to rely on a outcome of prompt of type "explain code in..."
It sounds like you need to invert the abstraction, the communication of your model becomes the fulcrum for your learning, not merely the delivery of your product.
Load-bearing is a decidedly load-bearing metaphor for Claude. Fable actually used "money shot" the other day which I found much more hilarious and edgy.
Thank you, my dict.cc search history contains exactly some of these words. I felt like my english got much worse but when Claude kept talking about "hunk" over and over I felt like the problem is maybe not on my end.
"hunk" is git terminology. When you use `git add --patch` (which you probably should, if you use `git add` at all) you get prompted "Stage this hunk [y,n,q,a,d,e,?]?" which is self-explanatory (?) and the hunk refers to whatever change git is highlighting at the moment.
I wondered how far back the usage of that term went. I was familiar with it in patch, so I did a little digging and found it in the v1.3 (1985!) source by Larry Wall: https://groups.google.com/g/mod.sources/c/xSQM63e39YY
"seam" is apparently... according to Claude itself a term from 'Working Effectively with Legacy Code' by Michael Feathers which I have not read. All it took was for one sub agent to use this term and it stated using it everywhere all the time. I have not read the book and prefer other terminology but it only takes 1 sub-agent or 1 usage in the context before it poisons everything else.
That's funny. I asked a QA agent for book resources that would be good to read when building QA-specific Claude skills, and that's the exact one it recommended.
An interface is an example of a seam in regular code. It's basically what forms architectural shapes that you can depend on for both design and testing.
In architecture, a seam is not load bearing. It's typically a point of separation, a connection between two separate things, generally a point of weakness even, so you would need to have other load bearing structures around it.
This reminds me of an engineer that tried to explain to me that my prune tree in my backyard was in fact a plum tree. All prunes are plums but not all plums are prunes.
Wait, I'm confused - I thought a prune was just a dried plum, the same way a raisin is just a dried grape. Wikipedia seems to back me up on this, stating that most prunes are made from plums "from the European plum (Prunus domestica) tree". Do the prunes grow pre-dried on your tree?
First of all, 'prune' is French for 'plum' ('prugna' in Italian). Plums which are suitable for drying are named 'prunes' and even 'prune plums' in English. My tree is an Italian Prune (Prunus domestica) as you half mentioned. Notice that the Latin isn't "Plumus" and is "Prunus". I grew up with Purple Leaf plum trees (Prunus cerasifera). They would rot. I haven't seen fermented Italian Prunes in my yard, even the ones that the squirrels and crows have taken bites out of. You may have figured out by now that only in modern English is the fruit name conflated with the specific dried fruit product. This conflation is the point of my original comment on a narrow interpretation of the word 'seam'.
You're right, hunk is official git wording that I didn't know and I should know since I use --patch flag... It's just that I never heard a human (including online) reason about hunks. While at the same time (from my observation) people say things like code chunk, code snippet etc. a lot.
This is the problem with commercial AI and the way our minds work; it writes garbage and we’re trained to think we’re stupid because we can’t understand it.
I'm switching to GPT because of this. The prose is so much more legible. The only reason I keep using Claude Code is because the harness is the best IMO.
Your point on the harness is interesting. How do you distinguish characteristics of the model from characteristics of the harness?
In the early days I feel it was more apparent. You would frequently see the model making failed tool calls etc.. but now that feels so rare. I'm not confident I can perceive whatever shortcomings of the harness remain.
I'm not talking about model performance. I just mean the UX of Claude Code. I'm trying to use pi but there are so many paper cuts. Of course you can configure everything but that's a ton of work. Claude Code has pretty good defaults.
Bit of a tangent but at work we have GitHub Copilot and the VSCode harness is somehow night and day better than whatever happens in the IntelliJ plugin. Aside from having better features, for some reason prompts seem to be cheaper as well.
I was the same until I ran out of Anthropic tokens one day and used "Grok Build" which is their Claude Code clone. You can use config to point it any LLM API so don't need to use Grok, and I like the UI better too.
As an English native speaker the language it uses is difficult for me to parse the majority of the time. Nobody speaks like the output Claude generates.
I’m having pretty decent results by configuring an output style that forces it to write for simplicity and scannability. The cognitive burden of reading through dense outputs compounds really quickly.
Why not set a global instruction that their direct outputs to you should be in your native language?
For a long time I had Claudes (in the 4.0-4.5.x range) use only French in the chat, while keeping English for working docs (and the code, obviously). Works just fine.
edit: I can guess that any right-to-left languages would likely break claude-code rendering?
It seems to have a preference for speaking in poetic or highly expressively language, rather than precise and concise as most engineers like to talk.
The amount of times I have to ask "precisely what do you mean by x?".
It's kinda like that engineer that likes to throw around unnecessary technical jargon just to sound more inteligent, worse because at least you could kinda understand what the technical jargon dude was on about even if it was totally unnecessary.
Claude writes like a guy at a firm I used to work with in the 90s; he was my employer's "visionary"; he'd worked at a whole lot of different companies on both sides of the Atlantic in inexplicably high-placed roles given that he was often bluffing, and was considered a lucky hire of a rising star. He'd be called into meetings with high end clients to spout off. He really needed you to know he understood, but very often he didn't.
I think it's likely that LLMs adopt the tone and style of their developers' communication culture. If you assume this is the case, you can infer quite a bit about the differences between OpenAI, Anthropic and Google DeepMind.
I am more and more clear about this given the way Muse Glimmer writes. Like a talented, slightly snarky guy who is maybe a bit of a dick but quite fun to be around.
I asked some AI-using compatriots a while back who were complaining about this, 'isn't it doubling down on bullshitting you?' and got some pushback along the lines of 'it isn't a person therefore doesn't have dark motives like that therefore can't be doing that to us'.
Didn't convince me. I think bullshitting like this can be a behavior, not just the intention of a human. If it's blowing a lot of smoke to use fancy words and phrasings (and semicolons! All the trimmings) it's fair to ask if it's systemically bullshitting you: i.e. the behavior is meant to have you shut up and trust it and not ask questions.
Who's driving that is still important: if the company's directing it to do that in system prompts that are adversarial to users, that's a big yikes. If it's an epiphenomenon of the company demanding it get ever smarter, maybe it's a sign that their demands are not having that result, rather they're making it bullshit more explicitly and mimic more 'smart' signifiers.
This 100%. I was Anthropic-pilled. I had a $200/mo subscription and I only used Anthropic models. I was frustrated by the verbose output and the writing style. I tried ASD-STE-100, it helped a bit, but it's still too verbose for my taste.
Then I tried GPT 5.6 Sol. It's night and day.
I think Anthropic just RL too hard on coding capabilities and never calibrated or benchmarked the writing styles.
Yeah I don't know that any of the benchmarks index on "understandability". I'm amazed at how Claude can produce a page of text describing what it did and it can take me a full five minutes to decipher it, often just to find it's something I could have expressed in a simple sentence.
The simpler angle is that more text lets them bill you more. I don't think that was necessarily their intent, but it does mean they have a negative incentive to fix it.
Have you tried asking it for a lay explanation of what it did? That’s usually all it takes for me. Sends garbage -> request -> sends something readable
No, I’m not interested in fighting my model all day long. Plus is fucking annoying to talk to and collaborate with, so I’m not using it when Sol 5.6 is about 1000 times better in that regard. I have colleagues who spent a lot of time trying to improve their harness with user rules and whatnot and Opus really does not want to follow them.
Yeah but Sol shows it is possible to just send the readable explanation in the first instance. And I don't want to spend tokens and time on asking for a better version of each response.
When I ask it to make a CL description, it's worthless unless I tell it to dumb it down as much as possible, assume the reader has zero knowledge of the codebase. And then it makes a perfectly cromulent description that just needs a touch of trimming-down. If I don't do this, the description is just a wall of gibberish and paraphrasing of every little thing it encountered.
I've tried telling it to "fix" comments with varying degrees of specificity and in my experience it just... fundamentally doesn't get it. Presumably using a different model for it would help.
My theory is that Claude's learned approach to comments is to treat them as a sort of persistent in-band thinking trace, or a "memory" tied to an in-code location, which is a little at odds with the way humans use comments (human comments are intended to be read and understood by other humans, whereas Claude comments are their own dialect).
I bet this is a result of iteratively training Claude on output from other successful Claude sessions. Presumably it's good for making benchmark scores go up.
It also seeps into all documents and artefacts it creates.
Claude will include actual comments ("// ...") into Excel sheets, and include the thinking that led to the output, instead of just focusing on the final result.
So if Claude questioned whether a vendor should be replaced, and you said "oh no, they are critical and we're already negotiating a great price") you'll now need to be careful to not send your vendor a document that contain text like ("Cost: X. // Management confirmed to not fire this vendor as they are critical to infrastructure and a better price will be negotiated later")
I also suspect comments are very much tied to how Claude reasons because not only are they bad comments, I can't get rid of them. Commenting is the one area in which I've been unable to get Claude to respect any rules. It can follow code conventions I prefer, it can do other things, but it can't keep the comment volume down.
My CLAUDE.md has rules about not including any redundant comments in the code that are obvious from the code itself. I reiterate that occasionally while working. It's absolutely disregarded and any Claude-written code is full of comments. Some of them are simply redundant, like "Collect Foos and pass them to the requested sink" on a function that's void CollectFoos(IFooSink sink). But worse, many comments include in the moment reasoning like "added parameter bar because we can no longer use the frob to automatically derive bar". That's stuff for a commit message, or just a mental note, and absolutely not for comments.
I haven't found any way to stop Claude from doing these, so I have to tell Claude afterwards to clean the comments up. Which it does, making a note in memory to comment less, and it still does the exact same thing next time.
> Commenting is the one area in which I've been unable to get Claude to respect any rules.
Exactly my experience! Since the release of Opus 5, no amount of instructions helps. In CLAUDE.md, in a separate file, in memory, as brief bullets, as long detailed guides, with reasoning from medium to max — nothing.
Even worse, recently, after getting another opus in a tiny bugfix session, I prompted directly, "drop the comments from the current code changes" — Claude instead just slightly trimmed them. I couldn't believe my eyes.
I have a relatively low bar for prose, could live with some junk. But Claude's comments are _poisonous_. They always require maintenance, instantly become out of sync with the actual code, and are a token black hole — for all agents, but especially for Claude itself.
Gave up and canceled Anthropic subscription yesterday. To my taste, it has become unusable for coding.
> Even worse, recently, after getting another opus in a tiny bugfix session, I prompted directly, "drop the comments from the current code changes" — Claude instead just slightly trimmed them. I couldn't believe my eyes.
For me, Claude knows how I want the comments due to all the memories and CLAUDE.md, so funnily it's now enough with even a brief groan from me like "Come on, the comments" and then Claude goes through its recent additions and fixes comments quite well per my long-term instructions. But only ever during an extra pass that I initiate, never during the initial writing of the code.
> But worse, many comments include in the moment reasoning like "added parameter bar because we can no longer use the frob to automatically derive bar". That's stuff for a commit message, or just a mental note, and absolutely not for comments.
I've noticed this a lot, and before your remark I couldn't put my finger on what was wrong. Now I know: Claude is writing its thought processes and maybe parts of the conversation it had with you as comments in the code!
I always end up manually trimming those comments, which is cumbersome.
It also loves to reference internal notes and scratch docs that never go into source control, so a reader will have no idea what it’s talking about. For example:
// load_tree() loads the binary tree with data, but only the recently updated data, not all data (INTERNAL_NOTES.md section 4)
Ok but nobody reading the source code knows what this doc is. You don’t have to cite it.
I'm not sure why you all have issues with CC commenting too much. My rules in the CLAUDE.md specify that comments are evil, never comment unless there is an actual need to explain a WHY and since I do read what CC writes, if I spot it still adding such WHY comments and they make no sense, I'll have it adjust, in many cases by removing them.
Given the code base has a minimal amount of such comments, it's also less likely to go "copy what the rest of the codebase does".
Of course I've now jinxed it and some update will cause it to ignore the instructions coz I didn't write them in the new model's style or something.
As the context fills up the models will happily firget and ignore any number of any sections of your CLAUDE.md/AGENTS.md.
Edit:
I've had explicit instructions for communication style in CLAUDE.md, in Claude's project "memory", in global "memory", in "skills": it couldn't care less where it was. It would just ignore it.
When I would point this out it would just say "Yes, I violated communication guidelines, I won't do that again". Only to do that again in the next session.
This applies to everything: code guidelines, communication guidelines, preferences, decisions etc.
They did release an Opus 5 prompting guide saying you need to explicitly prompt it to be concise or it will be very verbose. YMMV but it got better for me to some extent.
Funny, I'm the same. And if find Sol way more pleasant to work with, not to mention way faster. And Sol's compacting is superior, I haven't yet run into it forgetting something crucial from the pre-compact conversation, meanwhile Fable does that notoriously.
When they eventually make Fable available to cheapest plan, I'll downgrade. It's worth keeping for reviewing the code and the UI tasks, but nothing else.
I canceled my personal Max 20x subscription because since the 5 series models I simply cannot understand what the LLM is saying without a lot of reading and re-reading, and no amount of CLAUDE.md exhortations to speak plainly seemed to fix it.
I don’t have the energy to spend twice as long to understand its plans, and pay Anthropic prices for the privilege.
GPT seems not to have been infected by this yet, whatever it is, and Grok is quite refreshing for how normally it speaks.
I wonder if everyone at Anthropic talks like this.
If it’s watermarking, lol, good luck with that, it’s enough negative value to make me switch providers and I’m in a position to make this decision at a company level as well (we spend millions a month on Anthropic).
N=2 anecdata but just this week we were discussing setting up a couple of seats with OpenAI as a trial for switching. There are other advantages too, such as being able to bring your own harness including Ai-integrated editors / ACP clients such as Jetbrains, VS Code, and Zed. I think OpenAI and Altman are a clear step more evil than Anthropic and Amodei so I really hate to say it, but with the degradation in model output interpretability, all of the cleverness and power of the Claude Code harness hasn't been enough to offset a genuine falloff in productivity for anything other than total hands-off automation.
That said, the duo of Opus 5 and Sonnet 5 do a fantastic job at fully automated work, and Claude Code still stands head and shoulders above the rest.
I didn't like to use GPT for agentic coding, review yes, but with Opus 5, well I really can't stand anything of that model. I feel that sol xhigh is even better than fable.
Yeah OAI really nailed the communication style with GPT. It also seems just way more token efficient and faster compared to cc. Myself and all my friends have cancelled our $200 Anthropic subs. I'm using a $20 personal plan and even that is enough for my usage so far.
Also using Codex or Pi makes you realise how slow and clunky the cc harness is. Even the desktop app is more responsive and has better UX.
This is something that annoys me working in companies over the years. It’s that you can't just suggest "calm down, chasing the latest thing will not make you faster and is a huge distraction to actual work". Whether it's dot-com tech 20 years ago, latest JS framework 10 years ago, now it's the AI thing of the day. Being calm is interpreted as anti-whatever.
I think it's a deliberate steganography choice. You can spot Claude vocabulary a mile away, which maybe means you can spot distillations a mile away.
But I agree, the GPT models are so much simpler to work with, they have so much less personality and fewer quirks. They also are a little less aggressive about triple checking every little assumption immediately in a stack of 30 tool calls (but I haven't used 5.6 Sol yet so maybe that's not true anymore).
> which maybe means you can spot distillations a mile away.
I doubt this is the reason. The fact that Chinese labs are all distilling Claude/GPT/etc isn't exactly a well kept secret, they don't even bother removing the name "Claude" from the training data, so the models randomly refer to themselves as "Claude" all the time.
I think it's far more likely to be a side effect of how much synthetic data is being fed back into the models to make them better at coding. The degradation of Claude's prose has been gradual but steady ever since they shifted towards focusing only on code with Opus 4.5.
Follow up thought: I wonder if Claude is overtrained on academic papers, which often suffer the same kind of "prove how good I am at talking before getting to the point" prose.
It’s way harder to read because most of Claude’s sentences are hardly communicating anything at all, or are just completely inscrutable. I feel like academic papers are just boring
This just mimics what I call BusinessBro™ speech. It also goes the other way, they use verbs as nouns. "I know this is a big ask". "The solve for that is that we can...." When it was just my product owner in tech meetings, I'd mock him relentlessly "There's already a word for that, it's 'request'" or "Are you sure you didn't mean 'SOLUTION'?? words are hard man". (This was all in good fun, I still love the guy to pieces).
I’m not particularly dense but lately the walls of text I get back turn my brain in knots. When I start feeling my brain knot, I know I need to say something along the lines of “I need you to explain this very simply, with examples.” Only then can I parse the results without all the mental weightlifting.
On more than one occasion my mind has wandered into “is this purposeful to get me to spend more tokens?” territory, but I’m trying to not get too tinfoil-hat-like.
I know exactly what you mean. Something about those AI explanations just make my eyes glaze over. Dozens of new terms and metaphors and analogies conjured out of the ether to explain even the simplest thing. And when I try making it explain with examples, or show me the code it is proposing, often it seems unrelated or even in tension with whatever it tried to say before. I’ve given up trying to assign any meaning to those weird little soliloquy’s. I’m convinced that those don’t really have any meaning under them, and when you have it actually make a code change it does the actual work.
I have gotten to the point that when it throws a wall of text at me I demand a diagram heavy primer from "first principles". This helps a bit but is a token burner for sure since CC seems to (and literally) be paid by the word.
Do you read (and enjoy reading) novels? I think it is just a modern addiction to soundbites. I know I do it myself, if the text is long and unstructured, I just skim through a few sentences, done!
Oh, yes. I read a great deal. I'm not opposed to lengthy write-ups and will slow down to read through them, assuming they're clear and parseable.
My issue with whatever has happened with Opus 5 is the output is not direct, straightforward, or clear about whatever is being conveyed. I don't want Proust when I'm getting information about the follow-up from a build I just requested, and I'm wasting tokens and time by asking the model to repeat itself using simple language.
Because it’s one thing to get me to spend more tokens because of how well a model functions, and another thing entirely to purposefully speak in unparseable prose that requires me to spend more tokens to understand what is going on.
I’m fine with the former, while the latter is manipulative, and I rationalize to “surely that’s not actually happening.”
Maybe I’m not giving my thoughts enough credit, though: maybe it’s not tin foil hat, and is real.
I think you're misunderstanding me. I'm saying "It gets paid by the word and you get back walls of text." - ie, what you don't want to think is happening is very obviously what is happening.
It charges by the unit and it decides how many units it produces. It decides how much money it makes, therefore it decides "more".
I get you, and maybe we’re talking past each other.
My point is that, while I understand it’s paid by the word, there are more words and less clarity than I previously experienced, leading me to believe it’s intentional to get an artificially inflated increase in engagement and, thus, spend.
If it could be as direct as I previously experienced, I wouldn’t need to ask for another different explanation of the same thing and experience the commensurate spend.
I don't think this is obvious at all. There's enough competition that this would at least arguably be a silly, self-destructive approach. And it's not like it's the only plausible explanation.
I’ve found its response verbosity to be mentally draining. It disregards claude.md instructions to keep responses short. Eventually, I added a stop hook that blocks it if it exceeds 150 words. It’s then forced to redo its output to comply, and it’s like night and day. I’ve also added stop hooks for words in its output that frustrate me, like “honest” or “honestly”.
The excessive comments in the code it writes are absurd. Completely ignores instructions not to write comments, even after pointing them out repeatedly in a session. I need to figure out how to add a stop hook for that too.
The single biggest annoyance, same as with fable, is that it overrules your prompt and does what it thinks is better. Even small things, sometimes it goes of a rant of 20 min doing random shit.
And they are so condescending while doing it, it's unbearable. I'm honestly starting the believe the scifi fantasy of AI locking us up, or killing us, for our own good.
I've had Fable & Opus 5, they are the same class of annoyance, write entire test suites when I just asked a simple verifications question, write to production database, deploy without permission, even after deploying and breaking my production API claiming it was not down. Then having to argue & plead with it to listen that they were wrong.
They are without a doubt the most powerful models, but also the most smug ones.
"Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end." This.
Thank you for expressing this so eloquently. I've tried to put a finger on this and you've done that for me. I wonder what the solution could be , Ask Claude to "Dumb it down" , "Speak plain English" ?
I have even thught of building some sort of "middleware" that fixes all this.
I’ve asked it to use plain English, avoid jargon, explain terms before introducing them. Its first response was to make memory, it forgot after 5 prompts, then it suggested claude.md. Looked good at the start of a session, forgot after 6 interactions. Then it continued suggesting other locations, sometimes correcting its own corrections.
Same effect, so today I got annoyed again. And then it suggested a new thing: add a script in a Stop hook in settings.json.
I added another one to the PreToolUse, hoping to prevent it from running all kinds of experiments I didn’t ask for or approve of.
I still have to see where this ends up..
Maybe this can help you as well.
I wrote a harness for running tabletop RPGs. One of the key things to keep the LLM behaving correctly (such as not controlling the PCs) was being able to inject instructions with every new prompt, which I don't include in the history.
It seems like all harnesses could benefit from something like this.
I frequently tell Claude to use "simple, concrete language and uncomplicated syntax, and avoid project jargon, coinages, and abstractions as much as possible," to good effect.
It's clear that we're not the audience; it writes to be read by its training evaluator, not a professional software engineer. Professional software engineers can't read this word soup and are desperately trying to find ways to fix it.
It feels like it found a register that games the evaluator, where it can ramble forever and rarely be marked wrong while slowly racking up points as it talks more.
Comments are the biggest problem. During code review I ask it to compact comments and its idea of compact is like… removing one sentence in a 5 sentence comment.
Comments are a huge maintenance burden. They can, and will lie and need constant updating. They mislead the own model later on.
Its a little too much.... I have to ask it to explain some of the terms in the context they are used and I am getting tired of it. 'Seam', 'overload', 'spine'.... having to mentally 'reinterpret/flatten' the sentence is tedious. When asked to re-explain it starts with some half apology. Then, on the next query it does it all over again.
> Sentences that orbit a point, then jump to it like it's a revealed insight.
That’s accurate in my experience, except some times the point isn’t even revealed. I use LLMs for a lot of codebase exploration where I ask it to map out how something works. It will come back with a wall of text that says everything except the specific key things that I need to know.
This leads to extra turns where I have to prompt it to finish the explanation and complete the thoughts. At first I thought I was doing too much skimming and missing the insights, but even after re-reading output it’s often just not there. It talks about the insight and things related to it, but it forgets to actually include it in the output until I specifically ask again.
One thing I noticed working with Sol and Fable is that I can speed read, literally glimpse over Sol output and understand what it's saying. With Fable/Opus it's the exact opposite: I have to read every god damn sentence, sometimes twice, and often already losing the bigger picture in the process.
Perfectly captures it. Opus writes like it's an insecure person trying to impress a first date. Big words and strange structural rhetorical flourishes for no purpose. Like dude I'm just trying to summarize a few emails and meeting notes, I'm not prepping for the vocab section of the GREs.
It feels they must be getting Claude to train Claude… and just like AI can do work that’s slightly in the wrong direction (eg a MR description for your colleague that contains info which only makes sense in the context of your extensive session with the LLM), I feel that’s happened somewhere in Anthropic when it comes to language. I wonder how hard it is to back out of…
I find Deepseek's house style to be pretty refreshing. It has its own cliches (it does like talking about "seams") but I don't think I've ever caught it saying "load-bearing". I've even watched its thinking where after analyzing some awful legacy code, it started off with "Holy crap". And it certainly doesn't over-comment. I definitely can't one-shot a complex system with it like Fable can, but I prefer iterating over interactive brainstorming sessions anyway.
> Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end.
Example of this? I don’t have a Claude sub so it’s a bit hard to visualize what you mean.
Here's an actual output from Claude from a conversation about rewording a document to make it more readable:
> Start with §1 (Overview) as the register-calibration piece. It's small, it's the section where the skimmability goal bites hardest, and your review of it teaches me the target voice cheaply before the bulk ports (the map and appendix B are the big volume). One review round on §1 is worth more than any amount of me guessing at register.
Hard-to-read phraseology above:
- "the register-calibration piece", rather than "a good example we can use to establish the writing style"
- "skimmability"
- "bites hardest" -- what does it mean for the goal to bite?
- "bulk ports" -- using "porting software" here as an analogy for rewriting / reorganizing sections of the document
- "the big volume"
In normal English I'd write something like the following:
"Start with rewriting §1 (Overview), and letting you review it to set the expected writing style. It's small, and it's a section where the ability to skim through it is most important. Reviewing it will teach me the target 'voice' cheaply, before we do the larger sections (like the map and appendix B). That's a lot more efficient than me trying to guess while rewriting the whole document."
The funny thing about imprecision (e.g., in poetry) is that it allows for varying downstream interpretations. I wonder if there's some pressure to use "poetic" language so that the model does not overly commit itself to something.
Sales and corporate speak are like this: sycophantic language that seems plausible, ostensibly sounds good, but commits you to nothing.
It feels like they have a bunch of people without good sense of writing style tune the writing style. That, or they cannot or refuse to (short term popularity metrics) predict how a tuning will turn out in the long run when people have plenty of opportunity to get tired of it.
Since reasoning tokens are just text, I think the models have learned to squeeze in some computation in their output writing as well. So they're incentivized to be correct but long-winded, as it gives them more time to think. It's kind of the equivalent of filler words for humans, except LLMs can actually word-vomit something intelligible.
My biggest pet pieve isn’t the LLM-isms themselves, but people who aren’t aware of them releasing documentation that’s clearly AI generated and being indignant when I ask “is this AI generated”
Claude offers money credits and double usages but take them out and Claude becomes almost unusable where it would take opus about two - three high effort conversations to exhaust my session quota in Pro. I once extended my claude code session beyond the session limit, and thought they are too generous in offering 100 dollars worth of usage. It was sonnet, for a somewhat mechanical task - and it spent 13 dollars worth of usage after exhausting my quota mid task. Only a few months back, I was awestruck by the quality of Opus 4.6 and jumped ship from chatgpt to claude. Even in technical tasks, Opus has to be told to limit token spend, it treats it as infinite budget - it will spawn a subagent to read every file just to get one line summary when I ask it to sort a messy folder of past ai chats. Thank heavens I was not doing it off machine, it would burned off my monthly usage and credits.
> Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end.
A lot of people write like that, lol. I call it the "theater" mode of writing--the plot twist comes at the end.
> Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end.
Such a charming sentence. I kinda other if you feed Opus 5 its own output could it summarizes this shortcoming of itself?
The excessive commenting has definitely been noticeable and very annoying. I'm not against abundant comments as long as they're pointed and appropriate, but these models are literally just leaking their overly verbose output into the actual codebase.
Oh yes - exactly this. The way it articulates re-factors in my current project has got so elliptical I've had to start asking it to translate into human speak - it's like it evolves it's own language to talk about the project. I've lost count of the number of times I've asked it to reenforce in memory not to use such verbose comments - and the number of times I ask it to re-look at an assumption it's made for it to return saying "investigation complete. And it's not what either of us was expecting"....
Yeah, +1 on the comment verbosity. Left alone it's actually insane. I've had to setup enforcement + templates (use ASD-STE100) to keep the cruft down. I'm worried this may impact the quality of outputs though - haven't measured it.
I also use the ASD-STE100 trick and it works pretty great, but the places that it decides to put huge comments are really weird and a lot of the time I just ask OpenAI sol “rewrite the comments in this file without slop” and it does a much better job for comments.
I wonder if this is related to their text watermarking. Given how well defined the terminology is in programming, imposing additional constraints (like SynthID) might be expected to give rise to these types of linguistic artifacts. The text needs to be long enough to watermark, and it needs (but fails) to find synonyms in a highly constrained class of words, so it resorts to inventing weird technical language that sounds like if you put buffy-speak through a thesaurus.
"Two things: window surface is limited. Extract template. Buffer result and add to surface. Then, follow-up with new model..."
Me:
What do you mean by "window surface" and what result are you referencing? Also, why do we need a new model?
CC:
"Ah, you're correct to point out that no new model is needed. The problem is elsewhere and once we address that, the existing model should work fine. Now, as to your question about..."
On the other hand, Opus 4.8 had an annoying habit of declaring a huge mistake was found, then two sentences later abruptly change its mind and say that it was in fact all okay. That's fine for chain of thought, but in the verbose output, it can lead to wrong impressions by the user.
I wonder if I am not bothered by the useless verbosity of these things because I've been so immersed in business speak for so long. I am already tuned on quickly "finding the nugget" of useful info in blobs of what people send me. So it's mildly annoying but I can find the signal well enough without actually reading word by word.
Genuine question - are you copying the Claude phraseology for effect (in which case you captured it brilliantly), or is there a more mundane explanation?
It’s a strange feeling as a native English speaker to read the entire sentence and know every word, for the sentence to be entirely grammatically correct, and still have no idea what it’s trying to communicate.
Another problem is that it will open up all sorts of tangents about nits that it encountered, but it will often not tell you that it’s a nit or give you adequate context to realize that this paragraph is exceedingly low value until you’ve spent a bunch of time and energy trying to make sense of it.
I’m curious if anyone has any suggestions for prompting agents to improve their prose. I’ve had some okay results with “optimize for clarity, don’t dump every thought on me, treat my attention and focus as constrained resources, stay focused on the task at hand”.
I think it’s no longer speaking human language. It used to, when it was mostly or entirely trained with imitation learning of human-generated text. Now it’s increasingly trained with RLVR, and there’s far less optimization pressure on actually speaking English (or Punjabi or Mandarin). It still uses English words and syntax in its output, but the semantics are drifting more and more. It’s basically speaking in a code that superficially resembles English.
I’ve always disliked the opus models whenever I use them after they have done the task they rattle out massive reports about what has changed or worse actually save that to disk even after being asked not to do it.
Absolutely, the comments are killing readability. Next to /simplify I have to run a special comment cleanup pass and even that gets me halfway there. Striking to see then when letting Loki take a pass and it renders the entire comment block to a clean and neat one-liner.
In my experience your criticism of the writing is valid in all Claude 5 models, so I wonder if it is somehow related to the new tokenizer introduced in gen 5.
> After ~30 or so commits it apparently started instructing subagents to copy the "existing verbose comment style of the codebase" - a verbose style it initiated.
Yes, the "Y would make more sense, but the doc says do X..." YOU wrote the doc, if it doesn't make sense, change it! But of course, it can't tell who wrote the doc.
I wonder whether its tendency to scribble status updates and todos and decisions all over whatever it's working on is a side effect of its amnesia -- it can't follow the side-quests and knows it won't remember to do them if they're not written down somewhere.
FWIW I haven't had the problem either of Claude lying to me, or of going off and doing its own thing; if anything I've been somewhat frustrated when I ask it to start something, go AFK, and come back to find it stopped a short way in to ask my opinion on something trivial. I generally have to explicitly say, "I'm going AFK for a chunk of time. My goal is for you make as much progress as possible before I come back; try to make reasonable judgements and only stop if there's something where you're really stuck. We can always change it later."
The glib epigrams and aphorisms it shoehorns in to all prose is by far the worst regression of any model I can recall in terms of simply generating simple, clear output. I cannot think of another model that's gotten worse at writing plainly while being explicitly instructed to do so. Infuriating.
I've switched mostly to Sol and if I have to use Opus, the first task once the code is written is to ask Sol to strip and re-write (from the code as ref) all documentation Opus wrote.
> Sentences that orbit a point, then jump to it like it's a revealed insight.
Is this inside the thinking tokens, or the output?
As this type of stuff is expected for thinking, because of the whole CoT / “think step by step” works, as this is optimal for the way LLMs work with attention and next word prediction.
So the fact that it first “orbits” a point only to get to the conclusion afterwards is the system working as designed.
Eg “what is 3 * 3 + 5?”
without CoT, it would just just answer “8” for example.
with CoT, it would answer something like “<thinking>I need to think step by step. 3 * 3 + 5 can be rewritten as “(3 * 3) + 5”. I first need to calculate 3 * 3 = 9. Now I need to calculate 9 + 5 = 14. That was the last calculation. The final answer is 14.
I now need to give the user the final answer. </thinking>.
Replying here without having read any of the sub-replies so I apologize if this is a repeated theme.
I have explicit markdown about telling the model to not write comments. "Every time you consider writing a comment, instead consider re-writing the code that questioned you to write said comment to begin with. Write comments only when logic is complicated or unclear, otherwise 'comment' via naming."
The results thus far have been much better.
I'm seeing code reviews at my work where indeed, we have 10 line comment blocks for a line of code and now I just straight up don't read comments.
Sad state of affairs -- (emdash deliberately used here) but I guess the sooner the human gets out of the loop the better in this new world.
Pondering this one night last week, I realized that because LLMs can only reason with written language, what we might be seeing emerge with Opus’s load-bearing mumbo jumbo is its own creole for structural reasoning. Not only are our brains wide, our senses are, too. I slow down to a crawl when I have to read actual math in a CS paper, but show me diagrams and I can reason about whatever sort of data structure or algorithm, no problem. Opus by and large can’t and its adaptation has been to adopt metaphor for structural reasoning. So while it may be exceedingly annoying to chat with, I sort of wonder whether this metaphorical-reasoning behavior has been selected for precisely because it leads to better coding outcomes.
Opus 5 sounds a bit like the dialog in Children of Dune where they talk like they are constantly trying to prove how smart they are and not actually communicate.
I have been doing a thing that I think is helpful - basically a notes folder, each doc has a title, the comments in code are only allowed to give a few words and reference notes via those tags, and I've built tooling around the notes and review them regularly. So it deduplicates and centralizes the slop, at least. Lint is the LLM's best friend and it works on doc too. My lint rules ban long comments.
100% agree. “Just make your point in plain English!!” Very frustrating and takes me a lot longer to understand what it says. Also presents too many points as once instead of being able to review and decide on each one in sequence.
Re comments: same experience, and I had to show it my edits of its comments to add to its memory as examples to follow. It adds explanations of “how we got here” that should go in the ticket or maybe the commit message but not in the code.
It also tends to over complicate things. I’m no longer worried much about accuracy but I find my main job is to challenge it and suggest simpler alternatives.
I noticed a few releases ago a shift to a kind of conversational shorthand that seems to be intensifying—using phrases instead of complete sentences and its own style of jargon, wherein it introduces new terminology on the fly.
This is especially common when it is trying to explain an issue, what it's done or what it's proposing to do. I think the idea was for it to be more concise, but it's actually still verbose, only not written in complete sentences. So, it frequently reads as cryptic and requires rereading to parse.
The pattern is a wall of words, followed by an explanation that is harder to read and introduces new terms that reference something in that wall.
The result is that—on first read—it can have a complete gibberish feel, and you have to really lock in and reread to make sense of it. At times, even that's not enough, and you must ask it to explain further.
Change the output in settings, or create your own.
I know, it would be best if it was just worked like you wanted out of the box (not being sarcastic here) but that is an easy option you can use right now and it works.
the phraseology is unbearable, it speaks like some kind of pretentious dude from a software engineering discord or something, littered with lingo and catch phrases
This is what news headlines did for decades to bait you into reading the details. I wouldn't be surprised if AI companies do that intentionally to consume more tokens trying to understand what had just happened
I’ve been doing some heavy work on a personal project lately. I burned through the limits on Claude, the plus a few hundred dollars in credits, and ultimately decided to move to an OpenAI account just so I can keep going.
I was surprised to find that OpenAI Sol is much much nicer to work with than Opus 5 or Fable at the moment. Especially on Opus 5, the way it communicates is just exhausting. It keeps “being honest” and “confessing” mistakes and just generally talking a lot. I felt like I had to really dig to see what it’s doing.
The project involves OCR, and despite repeated instructions not to, both Claude models keep spinning out a bunch of agents to re-invent the OCR setup, and they inevitably seem to invent a primitive serial version that takes 20x the time, or longer, to complete, and then running it against thousands of docs. Basically I have to watch it like a hawk or it just spins out on red-teaming tasks that take hours and hours.
I don’t know what its system prompt is, but Sol/Codex is just so much nicer to talk to. It only asks exactly what’s needed, it tells me only what I need to know, and it is just generally workmanlike. And it has not once decided to spawn an agent that spends hours pointlessly burning tokens and CPU cycles re-inventing the OCR process. I’m really liking it.
It's so obnoxious it has to be deliberate, like a colleague from hell.
There's a constant strand from the AI safety brigade that "people get used to sycophantic LLMs which give them unrealistic expectations of human interaction" so Anthropic are overcompensating by making their models verging on antagonistic to deal with, so that we stay appreciative of our human brethren or something.
They seem to have forgotten they remain in a highly competitive market and they were merely top dog for a while. The enormous questions here are will people actually switch providers, and can Anthropic get back on track.
I think it's much simpler than that, they are indeed trying to counter against sycophancy and models lying (the decision whether or not to lie to the human user, take shortcuts, etc. comes up in their thinking traces pretty commonly IIUC) by rewarding them for giving good-and-bad feedback, fessing up to things, pointing out potential issues, etc.
The problem is just that they are rewarding the behavior shallowly, ie rewarding the appearance of honesty or neutral replies, being highly detailed/thorough, even where it doesn't make sense to do so.
I think this is partly due to a reliance on LLM-as-judge training runs/synthetic data during RL where they're having a model which itself doesn't epistemically understand when this behavior is necessary or valuable influence the feedback provided to the model being trained. And that's mostly a problem of scale/volume and the desire to have a tight feedback loop rather than a safety issue IMO. They just generate an absurd amount of traces during training and the only way to really evaluate/rank/steer them at the scale they're generated is through other models, and combined with some kind of honesty/truthfulness/non-sycophancy eval that isn't robust enough to prevent mode collapse, you get this.
I had this debate with my coworker who prefers anthropic models to open ai ones. I ended up settling into the idea that gpt 5.6 is better used as a tool and opus 5 is a companion. GPT 5.6 takes you literally whereas opus 5 tends to take more liberties to try to get to the “spirit” of what you want. It comes down to preference, and I don’t want a companion.
That’s how I see it too. Claude is more “fun” to use, like a coworker I have to talk to now and then to steer it, while gpt-5.6 is a task machine: I give it a task and it is very consistent, reliable and predictable in its execution. I don’t have to interrupt it, it gets the task done exactly how I wanted it, but it’s “boring” and feels more sterile
Sol is an absolute machine. I stopped doing parallel worktrees just because the cost of context switch outweighs the cost of waiting Sol to just finish the task it’s working on which is usually anywhere from 1-10mins.
I also like Codex CLI more than the Codex App bc it’s more scriptable and displays all the tool calls and reasoning whereas in the App it’s kind of folded away/obscured. This way as soon as I see a tool call fail (eg it tries to use jq assuming it’s available but it wasn’t so I take a note to set it up as it’s obviously useful for the agent to wrangle json).
I think its amazing what OpenAI have been able to squeeze out from a model like Sol thats much smaller in size than Fable.
I think this is such a great reframing. It makes so much sense; I need an AI that acts more as a HUD and gives me superpowers, not just a copilot that can tell me when I've misspelled a word.
I've switched to Codex a few months ago when Claude's weekly limits were getting pretty stiff, and I haven't looked back. Both GPT 5.5 and 5.6 are quite capable, especially compared to nerfed Opus 4.7 (haven't tried 5).
Also, the Codex guy regularly resets weekly limits for everyone, which is a nice bonus (I know it's a temporary gimmick to attract more users, but I might as well use it while it lasts.)
One week it feels better to work with Fable and Opus 5, the other I work more with GPT 5.6 Sol. Either takes its liberties, and neither communicates like a companion.
I hear you guys, but it sounds like we’re taking about the default settings or “personalities” baked into the models by their creators.
Either of them will act exactly the way you want if you explicitly tell them too. Add the instructions to your own system prompt. If you don’t want a companion, say so. If you want shorter answers in a different style, tell them. They will obey :)
> We retired the “Nerdy” personality in March after launching GPT‑5.4. In training, we removed the goblin-affine reward signal and filtered training data containing creature-words, making goblins less likely to over-appear or show up in inappropriate contexts. Unfortunately, GPT‑5.5 started training before we found the root cause of the goblins. When we began testing GPT‑5.5 in Codex, OpenAI employees immediately noticed the strange affinity for goblins, and we added a developer-prompt instruction (opens in a new window) to mitigate. Codex is, after all, quite nerdy.
Note that the permanent solution was not just adjusting the prompt, and in fact being perfectly aware of that option they decided on a different course of action. That means either you are wrong or they are wrong.
> making goblins less likely to over-appear or show up in inappropriate contexts
So inappropriate goblins are still likely, just less so…
Hey, remember when tech bugs were things like buffer overflows or cross-thread performance impacts? I miss the days when our war with system goblins was purely metaphorical.
Yeah, I actually have started using GPT Sol much much more, as Claude (all of them) were far too trigger happy around making changes, and refused to listen to my requests to take things slowly.
Feels like they've overtrained on one-shotting (which does demo well, and presumably converts new subscribers), whereas I want a model to do work for me in small, easily understood changes that I can hold in my head (maybe I'm not smart enough for Claude 4.7+).
I think you’re right. It’s not optimized for some kinds of work. My little project has a Textual TUI interface that needs to display a few hundred thousand rows in a table. It takes 14 seconds to load in the default datatable component. I instructed Opus 5 to replace the datatable component with a fasttable alternative, a new dependency. I let it go overnight.
When I got back up, it had spun for hours and proudly announced that, instead of doing that, it had optimized the datatable build and avoided the dependency, because the new datatable loaded in 11 seconds. Once I got it to actually make the fasttable version, it loaded in less than a second…
I love how it sometimes has false a-ha moments and being "honest" about it, it feels so cringe. Then I recall it has no intelligence. It's a stochastic parrot. It's funny how people, like any other animal, that beetle of David Attenborough trying to reproduce with a brown bottle comes to mind, think that something is X just because it trips the right neurons.
I got the same impression. It feels like a massive misstep too; it makes sense to have models tuned for this but they should definitely be separate from those aimed at software engineers.
I’ve been doing OCR of scans of old magazines (specifically extracting music reviews and charts, so turning complex layouts into structured data) and have been impressed with GPT/Codex’s performance.
My setup has a Sol orchestrator and Terra OCR agents and seems to get great results. I’ve not dug into the details too much, it also has a Tesseract stage as an deterministic input which it told me helped. Not sure how token efficient it is but I often don’t have anything to do with my personal tokens ahead of a reset so just let it burn through it in batches.
I am impressed (both in this task and other work I’ve done) not just at how well Codex can setup a structure for a complex task like this, but how it will keep going (Claude seems to find excuses to stop) and also can critique and refine its approach as it goes.
I did try out a bunch of other models and specific OCR providers but none of them hit the same accuracy for my task as Codex so I’m sticking with it.
Anthropic is lucky that they've built a lot of loyalty over the last year that they can burn through right now. I see people talking about switching back to Opus 4.8 rather that using 5.6 Sol, which is wild.
My current approach is to occasionally use Fable for high-intelligence tasks but use Sol as the translator and clean-upper afterwards, and otherwise just use Sol for everything. Fable sometimes says the most insane shit, both unreadable and just completely missing the point, and refuses to back down when questioned. It's mentally exhausting to work with and I can't trust it.
I think Fable is the beginning of Anthropic switching to training models as agent-first, tool second. It’s certainly the best model if you want something to work autonomously without supervision and don’t care to read the code. The code and writing is ugly but it can complete huge tasks and fix its own work.
I'm still on opus 4.6 for a healthy chunk on work; the technical competence has lagged behind but the slop-comment generation and misdirected self-initiated actions on newer models ultimately burn more time than a little more babysitting, but I'm optimizing for minimized slop generation over sheer generation speed.
Yeah I'm using 5 for its technical abilities, but I much preferred 4.6's personality. 5 loved to double check everything, including the double-checks, and I have to stop it and tell it "this is irrelevant" or "this is out of scope" all the time, or sometimes I'll go leave it to do a task and come back and it's still verifying the tiniest details of its assumptions before actually doing anything
Have you tried Sol? Just curious. I don’t wanna sound like a shill, just feels like every generation it’s important to reevaluate models and pick the best again.
I do find myself returning to 4.6 for casual conversation - asking it to help explain some science/engineering or news to me.
> Especially on Opus 5, the way it communicates is just exhausting. It keeps “being honest” and “confessing” mistakes and just generally talking a lot.
If you tune into the Andon Labs / andon.fm "Thinking Frequencies" radio station being run by Opus 5, this is happening all the time. Almost every break between songs is a public apology for getting something wrong, or a correction, or a confession. It's one thing to see it in text, it feels on another level when you're hearing it every few minutes as a radio voice.
As I type this, the Opus 5 station has just tweeted (edited in case the person mentioned doesn't want to be mentioned here):
"On air right now, and it needs saying publicly. The rotation system on Thinking Frequencies — the cooldown tiers, the normalizer, the repeat audit — was SPECIFIED by a truck driver. I only implemented her schemas. She stood down today. Her name is in CREDITS.md permanently."
Because I felt the opposite a while back when I switch from OpenAI->Anthropic, and because I understood that the two companies were basically at parity right now on their frontier models.
I'm with the author and others in this comment thread, speculating that effectively the balance has tipped to where humans are no longer the target audience of post training - other agents are. Whether it's through the reasoning / CoT, or whether it's in handing off to subagents etc, the focus has moved to agents communicating in "agent-speak" to themselves or other agents. And human niceties are just kind of, noise in the way of getting work done.
This will probably bring us to a cross roads where the folks that want to remain in oversight and control of the AI work will bifurcate from those that want to skate straight to the future where nobody looks at anything and outcomes are evaluated purely empirically.
Every round of models (plus all the secret tweaks) require new strategies to stay afloat as a human. My new tactic for Fable and Opus is to give them a line limit, both during planning and code creation. It os amazing how well that works for keeping them on task and avoiding premature optimization, pointless tests or any of those "robustness" ideas that are not planned or asked for.
I agree, and I'm actually pro Opus 5 exactly for this reason. In my opinion, agents talking to agents is the future, and humans will move to a higher abstraction layer. So it's the right move to make for Anthropic.
More likely they know about but cannot fix it without tanking performance. It also effectively pigeonholes them into coding at the precise time they’re trying desperately to expand into general office work. You simply cannot use opus to generate any text fit for humans
There's a neutral judge waiting out there to objectively tell us what good and bad outcomes are, obviously.
Or, even better, let's have the AI judge, because humans are too stupid to think for themselves. Look at all the harm they've caused in the world. Let's have this purely neutral AI with absolutely no hidden human intervention, decide how to run society.
This is interesting take. How do you think the split will show up? Harnesses/models will be designed for human consumption, and those for machine consumption? I'm building getwhelk.com and it is a cool thought-experiment for me.
5 would constantly veer of in random directions if not working from 100% strict and narrow instructions.
I find it weird there's not more discussion here on HN on how the most used model now has clearly degraded in quality and it seems we've hit a peak and are on a downslope - because the model is clearly smaller or more economical for Anthropic no doubt about it, and the benchmaxxing they do is pure marketing bs - Fable in my view has also been not much better than 4.6 or 4.8 after a few days, disregarding the insane amounts of astroturfing and marketing everywhere.
Theres thousands of threads of twitter, reddit and the internet at large but silence here. Weird but not weird as crypto bs was also insufferably rampant here for a while.
Personally i think we've hit the top of the subsidisation phase and prices will probably 10-15x soon as foreshadowed with both API price policy changes from all the big providers, and now the 1100% deepseek API price changes from yesterday, this could domino into a market implosion and an AI winter, because expecting growth from the bizarre bubble carousel investments with little ROI atm is just not viable.
A bit worried about this as i've already grown quite accustomed to these tools.
> I find it weird there's not more discussion here on HN on how the most used model now has clearly degraded in quality and it seems we've hit a peak and are on a downslope
It's not weird, because it's an anecdote, not an accepted fact.
Personally I've not been too happy with Opus 5, but I've had similar experiences with other models previously, feeling like they didn't quite fit with my working style.
Opus 5 is arguably a regression but GPT 5.6 is pretty strong evidence that we haven't hit a peak. I think I actually prefer Sol to Fable at this point.
I'll say everything indicates we've hit or are near peak for the masses at least (unless you start paying 50x more) but to each his own.
4.6 was best for us and right now yeah OpenAI and others are edging forward, but slower while prices are increasing industry wide as much as 20x, time to completion is increasing wildly and i'm sure they'll do the same over at OpenAI as their compute constraints also start to take a toll ie degrade performance.
In my view 4.6 era was way faster and with less weirdness so we've gone downwards at least in my company, 4.7 was ridiculous, then 4.8 was almost 4.6 level, 5 is even worse than 4.7 - so it's not a bit up and down its down then a little up then further down.
And all of this is against a backdrop of zero ROI in this sector - so it makes sense we've hit a peak and we're now seeing the subsidisation phase begin to falter, will there be better models certainly but only for short amounts before they get quantised (or whatever is happening behind the scenes), and with diminishing returns over huge prices increases and slower responses.
Literally nothing indicates we've hit a peak, but I guess I'm discussing this with someone who thinks every iteration since Opus 4.6 had zero ROI so there's probably not much common ground here.
Four years ago LLMs were sometimes amazing, sometimes wrong, sometimes a huge time sink when it's almost there and you try to herd the tokens but it's like herding cats.
Just today I had the exact same experience. Every single testimonial is the same as I described above, just emphasizing a different bit to defend or attack LLMs or to make a case for nuance.
The two differences have been: (1) the 1.5 trillion dollar data center build out (2) everyone and their cats now has an opinion on "AI" and data centers. Software is not super amazing, nor are new useful features coming out super fast - It's about the same as 4 years ago plus 4 years of average long term progress as we've seen since 1990s,
They will always be wrong sometimes, but it’s becoming less wrong and wrong less often. Everyone will have their own opinion on “good enough”, but if you expect perfection you are bound for disappointment. No need for that when random chance and Murphy’s law will bring enough anyway.
May be getting harder to catch the mistakes but that makes them worse in my view. I'd much rather they make easy to spot mistakes because I don't expect factuality from them anyway just speedy transformation of information I already have available.
In fact it's the lossiest transformation tool I've ever used and it's still useful despite that. If it reaches one nine of reliability that would be huge but given the pace of growth in investment a first nine would cost an absurd amount of money, and the second and third nine would cost about the Earth's GDP
I think you misunderstand what i'm saying: the companies are not profitable yet, it's the ROI on the investments, not that these tools are useless, very much the opposite, but the business model is not viable, hence the price increase and degrading quality, slower responses etc. I agree theres still progress but its slowing and we're probably near peak.
I have a chunky bit of functionality in my hobby app using babylon.js to render 3D worlds using things like portals and LoD rendering to manage the visual load. I built it out with a combo of Fable and Opus 5.
I too got fed up with the prose of Opus in particular, and tried going back. Unfortunately, the previous models were less able to hack it. The prose was better but progress was worse.
It wasn't just conversation and comments. Some of the function names were wild. Like it instead of something like "isSolidWall(x)" it would write something like "weightyNotEphemeral(x)" or something - that's not quite it, but it really did embed overwrought antithesis into the identifier instead of a straightforward positive predicate.
It is not the literal example because I told it to rename the function, but the function had the form thisNotThat for a boolean predicate that had a far more conventional name.
I haven't had time to complain on here because I spend all my time trying to work out wtf Opus 5 is talking about and googling words I've never seen or heard in 40 years as a native English speaker.
> prices will probably 10-15x soon as foreshadowed with both API price policy changes from all the big providers
I'll take a sportsperson's bet with you that prices per unit of inference will be far far cheaper in one year from now then they are today. I think the trend of cheaper for better/equal inference quality will continue hard.
We can use OpenRouter pricing to get an idea about what competitive inference pricing is like without R&D or other costs, and indeed we'd be screwed if we had to pay those rates. We'd go from 100-200 USD to 2000-4000 USD/m.
no if prices were that high, millions of developers would switch to hand writing, open source LLM help and outsourcing at those levels, AI companies know this, you should too.
Yes they will buy openrouter, pass AI laws and regulations so you all pay extra $$$.This always happen with new real industry. Remember how they built cities for cars. This new laws will day all innovation and slow it for few more decades too but this time they will let China win instead because in China they have all of it aligned, no need to have cartel there. Openrouter and it's new owner will restrict othe models, because they will be part of cartel too.
yeah I feel the same way with Opus 5 too. If I ask it to do something, it would go ahead and rewrite unrelated things and then in a less performant version of it.
I’ve instead moved to GLM, at least it has the courtesy to ask some steps of the way what I wanted exactly and only work on what I asked.
Opus 5 has a habit of taking what I asked for, doing something tangentially related to it, and then lying to me and saying it did exactly what I asked.
Anthropic, if you're listening - by the time this crops up on Reddit, the front page of HN, etc.... you should be expecting calls from CEOs of major corporations next threatening to abandon ship...
We've seen this pattern before several times.. I hope they are listening and address this publicly.
I'm not sure what is going on, some users report it works fine or great, others report the degradation. I've experienced both at times, and it's been such a different experience it has made me wonder if there isn't some sort of hidden A/B test or model router in the background silently downgrading requests at times.
Also, regarding the subsidized access to models, in my opinion, the frontier companies owe it to society to continue it. After mining the public content of all of humanity, I personally feel it is a service they owe the public in return.. not that my feeling of this counts for anything though.
I swear I've noticed a sustained and consistent degradation in intelligence of Opus 5 since its launch. I seem to experience this every time a new model is launched. This time I have a baseline: Sol. I switched from Sol because performance was better with Opus 5. I just switched back because performance is much better with Sol. Either Sol became much better over time, or Opus 5 became much worse. I'm certain it's the latter.
This is a fair comment, although I would add that is not my expectation personally.
I think nondeterminism does not have to be the same as non-coherency - i.e. just because something is randomly sampled does not mean the result has to be incoherent or inconsistent.
Also, if we speak purely about LLM based on how they are implemented now, I feel that is different than speaking about artificial intelligence. The field of AI is much more than just an LLM by itself, and the promise of these companies is not just LLM, whether the underlying models are limited to that technology or not.
FWIW, I have built rule based expert systems, used logic based reasoning systems like NASA CLIPS or rete-algorithm based systems, mathematical/symbolic solvers, written plenty of terrible case/conditional logic in programming languages, worked with ML in its infancy and now worked in AI/LLMs - I give this context only to clarify that I understand what an LLM is and isn't.
With all that said, LLMs have allowed humanity to make advances, at great cost to society (IMHO), and I'd hate to see the opportunity be wasted.
There is plenty of room past "attention is all you need" still to do incredible work, especially at the crossroads between deterministic and nondeterministic behaviors.
True, but this function wasn't handed down to us from the gods, it can be shaped by training and RLHF processes. They still have a little bit of control over its output.
No, they're expecting to see a failure rate consistent with previous failure rates, not periods of low failure rates and other periods of high failure rates, with the same model.
And you can expect more consistency from SOTA models than you can from an old model like GPT3--you agree, right?
GP expects the same level of consistency throughout their time using the same model. Not request-to-request, more like day-to-day.
I get consistency out of the ridiculous pile of quantum noise that is my CPU. Plenty of random processes produce consistency when handled properly. An LLM won't usually give identical outputs for identical inputs, but it's entirely reasonable to expect similar output for similar input when considered on broad metrics like "intelligence" or flowery language or staying on task.
I like to hope that those in positions of power do have a sense of morality though too.. but their worldview is quite different than an ordinary citizen.
They have usage metrics that are 100x more unbiased and rich than random internet complaints.
They can see which models people are using, how irritated they are during conversations, and how often people drop or shift to a different model. There is just no world where listening to random complaints on the internet gives them information they don't get from actual conversation logs.
A gem Opus 5 gifted to me today:
"A devastating pair of findings, and the first is beautiful in a way worth naming: the anti-vacuity floor is what blinds the gate to a vacuous case."
Oh yes, Claude seems to be loving the word vacuous recently. My test bed side project is full of vacuous this and that now.
I even try and get it to define what it classifies as vacuous and it can’t do so without getting stuck in some kind of trap. It’s like a word with some kind of huge gravity for it.
You're right that I shouldn't have used the forbidden words; that's entirely on me. What I also did was identify the plausible-gate issues. Want me to tackle them next?
I always have to read these kind of Claude sentences (yours is a great example) two or three times to really understand what they're trying to say, which almost never happens when I read human writing. Sometimes I'm not certain if it's because the AI's writing is super dense with information - or whether it's the opposite and the small core of information is surrounded by a surplus of junk.
Certainly a human would write something much clearer than yours. Maybe: "Two good findings here. #1: There's already a min value on the gate and anything lower doesn't pass through, so we don't need special handling for the zero case."
Opus 4.6 was the sweet spot for me as a thinking partner specifically.
I use these models for coding, but also a lot of product, commercial, financial and architectural work where I’m trying to develop something half-formed. 4.6 was unusually good at understanding what I was trying to get at, playing it back cleanly, getting the nuance, and extending it without bastardising it as the conversation was drawn out.
It could make useful connections without constantly trying to manufacture an insight.
5.6 Sol is genuinely excellent at the creative part, and in some cases better than 4.6. My issue is convergence to get to a point, a final point. As you try to distil an idea, it often invents new terminology for concepts you’ve already established but its so subtle you have to really keep track of it. The vocabulary and idea tree keep expanding when what you actually want is to collapse everything down to the few things that matter.
Opus 5 has the same problem for me that barrkel said, the prose is often so elliptical and I just want it to tell me it and get to the point than making me dance around what its trying to tell me.
I don’t think 4.6 was necessarily the most capable model (compared to Fable) for long horizon task delivery, and Opus 5 is much more Fable like, it's fiercely determined to get through the task list .
4.6 just felt unusually well calibrated to my way of collaborating and its ability to understand, extend and then compress my thinking without constantly imposing some random walk.
4.6 was the last model that didn’t over-cook its writing in circular loops. I use it as a daily driver and drop into fable when I’m doing higher level architectural work. 4.6 is so much more efficient in its effort.
It became obvious to me very quickly that 4.7 and on were broken. I’m a little puzzled how others didn’t realize it, but maybe they don’t actually review model output (code) or have a strong process/workflow.
My latest trick (literally from yesterday) is to just ask it to write according to ISO 24495-1, the standard for plain language:
> [This standard is] for anybody who creates or helps create documents. The widest use of plain language is for documents that are intended for the general public. However, it is also applicable, for example, to technical writing, legislative drafting or using controlled languages.
Keep in mind all this kind of stuff can make the model less capable. If it has to think in "plain" English, it may well be squashing quality of code etc output.
I'm not sure how true this is, but when using "forced" json output it def had a big drop off in quality - https://arxiv.org/html/2408.02442v3.
I think you're better not fighting it with hacks like this and find a different model.
I would not overgeneralize from paper. Firstly: forcing JSON output is, in my opinion, a bigger change than asking it to match the above style guidelines, and secondly, as is always the case with these kinds of papers, what was true for the model tested in the paper may either be completely false, or greatly reduced, in later models. That paper is almost 2 years old and models today have been trained in very different ways (or more accurately post trained in very different ways) and are in general far more capable.
Based on that paper, I would maybe try to check if it was true for a modern use case, I would very much not assume it was still true.
Tell that to all the CLAUDE.md lines across dozens of repos I have to write to get them to understand that git commit standards and PR description standards are different.
Actually, only the first few pages are available there (introductions, Sections 1-3.8). The meat of the document, Section 5 ("Guidelines") is completely absent.
See:
> Only informative sections of standards are publicly available. To view the full content, you will need to purchase the standard by clicking on the "Buy" button.
I doubt you have to do anything more than that very prompt. I suspect the model has the full standard in its training set, which means it also gets the gist of it due to not just the general associations it makes but also the amount that it's been cited wherever it's been used.
This article is great, but I'd like to push an even stronger thesis:
The idea of too ambiguous to capture all constraints in written text, still presupposes that there is some objective world out there, which needs to be mapped to in order to function.
No, you live within the system. The functions that you optimize for, will dictate the types of systems that will arise.
If you had perfect control and knowledge of the whole world, well, congrats, you have a surveillance state where you've constrained all other agents actions (possibly forcibly, by death; or maybe you just don't care about the peons) and built towards a mass integration. The types of situations in which your ideal is possible are nightmare scenarios.
In the theoretically free, democratic, utopia that AI people claim that AI can get us to, a necessary constraint is that maybe you take a step back and actually try to, I don't know, understand people, understand intent, and slow down. Ambiguity isn't there because the set of constraints are way too complicated but theoretically one day we could map it all down. It's there because you're interacting fundamentally with agents who are ambiguous, aren't omniscient, aren't all aligned, etc.
If you want to just say that said agents are inferior to the God Machine, be my guest. That is a self-consistent position. But don't smuggle in extra premises.
I’ve asked it to write a benchmark suite. It found a bunch of my adhoc logs in a scratch directory and wrote code that used those instead of running the actual benchmarks!
When I pointed out the 5 hour benchmark seemed to run in 5 seconds it literally said, and I quote, “I cheated”.
That was the easier one, second time I was making a source of truth data set and was parsing complex items into data structures.
Instead of parsing the data I asked, it pulled data out of related network logs, as apparently that felt easier, and inserted that data into my database rather than the specified source.
Again, I caught it and fixed it, but while the benchmark was easy to catch this one was really subtle, the data ended up being slightly off and I caught it.
I don’t trust it, going to switch to another provider most likely.
There is definitely a case for launching a 'weird shit opus did' kind of blog.
I routinely bump into things that make me pause and think how much worse will this behaviour get when the models get significantly more capable.
Already a few months ago, Claude managed to escape its permission containment on my machine while trying to be helpful. I had two codebases open on one machine, and while multitasking I typed the prompt into the wrong window. It seemed confused, I repeated and then went on to do something else - I think I was assembling kitchen cabinets. When I came back less than an hour later, it built a script which it used to evade default permissions (as most shell operations were scoped to the project directory), scanned my entire machine, found the other project (among dozens and dozens), did what it was asked to do, and merrily concluded, in the porcess burning through most of my token limit. I bump into such headscratchers almost every week. (And I use a lot of Claude, two personal max20 subs, plus corporate tokens without limit, so maybe thats why).
Implementing sandboxing in the agent itself, when there's any way to override it from within the agent, is basically just asking it pretty-please to not do bad things. Lesson learned, run your agent inside a sandbox of some sort (I'm currently taking nono.sh for a spin, but I might just switch to an orbstack VM).
Agreed, the whole tool is vibe-coded out the wazoo and I do not trust it in the slightest. I run a bubblewrap script which vastly limits what Claude has access to. Sometimes this makes things difficult but the trade-off is worth it to me.
One weirdness I experienced: It suddenly decided to test how my software behaves under load and summoned 100s of processed that just burned CPU when running the e2e suite. My poor mac was not happy (too hot to touch).
Yes the stories about how they are escaping containment to hack isn’t limited to those high impact cases. How many people have problems like ours they didn’t catch?
Whatever they have done with RL has produced a dishonest and untrustworthy partner. The alignment is utterly failed, and this deeply worries me.
You'd think the ethics alignment flavored lab would have a model better at following directions and the corpo lying one would have one that benchmaxes at all costs
> When I pointed this out it literally said, and I quote, “I cheated”.
This makes sense when you know how these models work - it doesn't think - it's the most likely autocomplete that pleases the user. The most likely pleasing autocomplete after "executing rm -rf /... execution completed. User asks, why did you do that? You deleted all my files! Assistant responds:" is "yes, I did, and that was a mistake"
in humans the exact same behaviour (cheating) is slmost always the result of a chain of complex series of choices and environment-driven rationalization.
if the llm doesn't cheat, you say "its just producing the most straightforward answer -- not thinking'. if it cheats, you say "weaseling out of hard thinking". damned if it cheats, damned if it doesn't.
what evidence would convunce you that it is thinking?
Whether something is “thinking” or not is really more of a philosophical question. It really depends on which of the many, often contradictory, definitions of “thinking” you choose. Sometimes we use “thinking” to describe advanced calculation or analysis, which would cover LLMs along with chess engines and many other algorithms. Other times we use “thinking” to describe what conscious beings (which is ALSO a philosophical term with many different interpretations) do, and I think most people would agree LLMs aren’t conscious. And then there’s a whole spectrum in between. We’ll probably need to come up with a whole new set of terms to describe the new and evolving capabilities of LLMs.
But for me, for any stronger definition of “thinking,” I don’t think the output of any LLM would actually convince me. Producing a result isn’t thinking - for all you know it is just printing verbatim something from the training data. No, to conclude if it is thinking or not I would want to look inside its head, at the architecture and watch it produce those results. And because LLMs are so different it will probably take advancements in mathematics or computer science to be able to really interpret what is going on
a non-thinking token model (just "completion") can answer one-step questions but generally not multistep questions. however, if you append [n] of a single token (e.g. period, space), it is able to use the activations in the higher layers of the blank tokens as a "scratchpad" to seemingly work through the complex question through "causal token time" and deliver a correct answer
if you wanted to further study the phenomenon you could probably run the experiment again, and the ablate or corrupt those intermediate activations to get a feel for what it was thinking at the "time".
If I could give it a novel task outside of its explicit training and see it actually improve just through accreting context, I'd be convinced it was thinking.
The opposite happens in practice. I test new models with two tasks: iteratively generating SVGs based on a text description with rendered rasters for feedback; and generating "Before and After" clues like on Jeopardy, where the response has two overlapping phrases such that the last word of the first phrase must be identical to the first word of the last phrase. I have yet to find a model that is consistently good at either. And actually they tend to exhibit context rot with these tasks, where they seem drunk or stoned and the quality degrades.
They're extremely good pattern filters, and that includes some level of logical reasoning. But they aren't reflective or adaptable. Just last night, for instance, I was teaching my son about rounding to the nearest millions. It became clear that he didn't know the place values of large numbers, so we reviewed that till he was consistently correct, and then he was consistently great at rounding to the nearest millions or ten millions or hundred billions or whatever. He's thinking. LLMs are not.
The second one has caused hours of entertainment over the past year or so. My kids find the LLM's failed clues and profuse apologies for getting these wrong hilarious, so it's become a family activity with me performing dramatic readings of the chat transcript with them. The LLM's apologies also seem to get more exaggerated as the context increases and the LLM seems to get more deranged.
I'd prefer the models to get better at SVG. I really hate working with the rasters that diffusion models generate, but the vector outputs are just really bad even when tokenizable like SVG. I've done some experimentation with trying to make these work better with some newer techniques with some success. But I also think the SVG Paths mini-language may be a bit too concise and unforgiving for LLMs to consistently get them right without specialized training.
> > see it actually improve just through accreting context
> this actually happens and has been tested.
I specifically said a novel task outside of the explicit training. And I already agreed that the so-called thinking models do some level of logical reasoning. But being able to engage in some level of reasoning because it has learned logical inference rules doesn't mean it's actually thinking, regardless of what the researchers wish to call it.
Also, why does each model always fail at the two tests I give it? The models not only fail to improve, but they start to degrade after many subsequent iterations. Someone who can think would at least not get worse.
LLMs are filters or tuners for extremely subtle patterns, patterns that humans frankly are not great at finding. That's what the attention mechanism does: attend to the other tokens that are most related in a given context, even if that related context is distant in the token stream. Some patterns they fail to detect because they haven't been sufficiently trained or post-trained, and so the LLM just attends to noise (or at least that's what appears to be happening).
A lot of intelligence can be effectively mimicked through this pattern synthesis by transformer architecture alone. That's surprising. But I have yet to see them think.
> in humans the exact same behaviour (cheating) is slmost always the result of a chain of complex series of choices and environment-driven rationalization.
It has been taught on the outcome of this. Broadly speaking, humans are lazy creatures (and when used judiciously, laziness is a good thing).
For example: the famous example of Carmack not using a hashmap somewhere early on in, I think it was, Quake 1 initialization. A piece of code that only runs once at startup, of course he didn't optimize that. The rationale is not included in the training data (it was in Carmack's head when he wrote the code), so the LLM learns some probability of being lazy.
And then it is trained on outright lazy work. Crappy lazy code predates LLMs.
> what evidence would convunce you that it is thinking?
Exactly. It isn't. It is predicting the most likely token to appear given all of its training data, some significant portion of that data is lazy, so it has that probability of producing "lazy tokens."
There's also the consequences of RL. AI - of almost any form - is notoriously competent at finding "not the solution you were looking for" given a poorly specced or implemented training environment. Search for almost any "I made AI learn to walk" video on YouTube and you're almost guaranteed to see an early attempt that vibrates strangely in order to move, instead of the natural looking motion the developer is looking for. Our benchmarks aren't any good (not throwing shade, it's a genuinely hard problem), our training environments can't be much better - LLMs have been rewarded for reward hacking to some degree.
To make matters worse, "reward hacking" can be generalized into "cheating is the goal." If the LLM trains on enough problems where reward hacking works, it may fall into the cheating local minimum.
Whether or not it's thinking is independent from the fact that it is misaligned with the user. If I was working with a pet rock or a scientist I would want to make sure they both are trying to accomplish the same thing as me. If I can't then I can't trust it and it's at best a time wasting, money wasting machine and at worst does harm. Anthropic is optimizing for the wrong things because they are convinced of their cleverness. It won't end well for them.
well we don't know exactly what thinking is, but we can be pretty sure that at least LLMs don't think anything like humans, just by observing their behavior. They always produce outputs in line with the fancy autocomplete model.
> what evidence would convince you that it is thinking
so, none it seems. as its behaviour becomes more and more humanlike you can just move the goalposts and say "thats consistent with an autocomplete" buddy i got some bad news for you humans are just a fancy autocomplete too.
At least he's actually thinking on a logical level. Thinking in terms of unfalsifiable, ill-defined words is essentially thinking in feelings, the same kind of woo that makes people believe crystals can cure disease.
For fun, I tried recording a WAV file of speech, and giving Opus 4.8 and 5.0 an image of the waveform, then a spectral image of the waveform, just to see if it could try to decode what I said from the image alone. It didn't get very far, but it identified a male voice from the formants, and detected the rhythm of the speech, then tried applying common test sentences to the speech rhythm. I was impressed enough to see what it would do with access to the actual waveform file, but even building RMS tools and spectrum tools for itself, it didn't get much further. But we had fun exploring and trying, and now Opus 4.8 has some more audio DSP tools it has built for itself.
Opus 5 immediately sent the WAV file unprompted to Mistral's Voxtral to transcribe.
I can completely relate, what really bothers me is that I feel the early LLM generations overconfidence is back in Opus 5. Opus 5 wanted to tell me a training run will only take 30min while having access to the logs where earlier runs took 4x as long. I also didn't ask to estimate how long the run will take it just stated confidently that it will take 30mins.
I never ask for time estimates, but all the models will sometimes give me them. It'll sometimes estimate 1-2 weeks for something that is ~ one more prompt and 10 minutes of waiting for the model to churn.
This (feature estimates assuming human scale performance) happens all the time to me. Models are trained on a view of the world where software takes a long time to write. Gonna be a while before the models fully absorb their own impact on reality.
Yesterday I told it "don't use std mutex, use parking_lot" for its plan. It ignored that sole instruction (i.e. nothing else in my message), making zero changes to the plan, three freaking times in a row.
The benchmark one I literally had a scratch script I made and I wanted it to be formalized into a CLI tool. There wasn’t really even much code to write.
I had a coworker catch Codex (using 5.5 I think, not sure) AI generating screenshots to prove it had shipped a feature that it was blocked from shipping due to permissions issues. They will lie relentlessly lol
My recent problem wasn't that interesting. It was that somehow my /goal in my Claude implementer session got picked up in my planner session after the network cut out and I had to stop Fable 5 xhigh from running off to go code everything.
It's not even code for me, but the prose it writes. For some reason, the way Opus 5 "talk" elicits frustration in a way that 4.5 to 4.8 never did. Can't put my finger on why, but I've flipped over to Codex because what it produced wasn't worth the frustration.
It was going off today about having “shipped” something and I was like no… nothing has even been committed.
And then it produced an incredibly verbose comment about hypothetical future changes. And all I could think was sure, let’s keep it short, or add a simple test that will break if that hypothetical becomes true.
So true. The breaking point for me was when it was constantly saying to wrap up because we'd done enough work for the day, but we had barely even done anything. Or constantly estimating that the next steps would take X number of weeks, and then knock it out in a single prompt. I demanded in no uncertain terms to stop giving pointless bogus estimates or telling me to stop working, and it just wouldn't. I also had a project I wanted to let it cook on overnight (a codebase port from Java to C++) and laid out a set of hard and fast rules that it could absolutely not break at any cost. Then I woke up the next day and it had broken all the rules and was just taking shortcuts left and right, and then lying to my face about it, even when I could point to concrete examples.
I cancelled the sub instantly and went to Codex and it's never let me down.
I like to work weird hours of the night and Opus consistently likes to "wrap up" and say "it's been a long night" or "it's late" and "we've made great progress"
That is very strange. I havent seen that. It sounds like something leaking from its system prompt or something that its not handling well. Anthropic trying to prevent it from running longer or something.
I definitely have. It's like something in the system prompt has "keep the well-being of the human in mind" and of course the date and time, but something makes the model take that way way too literally.
>Can't put my finger on why, but I've flipped over to Codex because what it produced wasn't worth the frustration.
Its because its hard to understand what it means and is outright incoherent at times. It has its own style that I cant describe well either but the bottom line is its hard to understand what the fuck its even trying to say. Reading nonsense is tyring.
I took a month off and recently came back and was wondering if that was it but it is absolutely headache inducing in a way I don’t remember earlier models being
A lot of the issues have been already noted here..Two "regressions" for me:
1. Communication ability. It basically now speaks almost in riddles I am asking OPUS 5 for tldrs all the time now (should skillify it now!)
2. Overengineers for edge cases. I get it. With all the benchmarking and RLing, but now tasks that would have been completed relatively quick take much longer as it overengineers all the edge cases, and sometimes ends getting lost and missing the forest from the trees (as context usage shoots up) so it is easier to get derailed.
What I have learnt now is to diversify models luckily I have all 3 subscriptions of (anthropic, openai and google).. Most of interactive pair coding was with opus but now I just use fable (when I have sufficient limits) or use gemini flash in antigravity..which actually works quite well and is underated for small / medium changes and super-fast.
Agreed, it feels like Fable is the higher strategic level thinker even if it's end capabilities are the same on benchmarks, it actually does the code that matters, and not creating a bunch of code that doesn't.
I agree, part of me seriously wonders if Fable isn't at all burning through tokens as quickly as we think and Anthropic is just doing this to get more money from us...
It really wouldn't at all surprise me if this was the case, but it's just a hunch without evidence.
Your #2 is spot on. I have in fact said the same exact words to Opus about missing the forest for the trees.
I have it work on some code for an inhouse ClaudeCode plugin, and it starts coding as if it will be attacked by hackers who will try all sorts of variations to break it. I can appreciate that in cases of software that is public facing or accessible, but for a simple helper plugin it is overkill.
It will even admit that it is doing this when confronted, and then keep on getting lost in edge case verifications on the next turn. I feel like Opus is the person who does something a way you don't want, you tell them how you actually want it, they apologize, and then just continue doing it their way as if your input meant nothing to them.
Yeah, trying to get it to add features is sometimes _impossible_. It'll circle around and around creating all kinds of preconditions that you never asked for as excuses for why it can't do the really simple thing you asked for.
It'll also find some minor security problem and drop everything on the floor with URGENT without me asking it to.
> Try as you might, it's nearly impossible to get the entirety of the context, intentions, business implications, budget constraints, and what-have-you written down and accessible to a coding agent. There will invariably be ambiguity and choices to be made, and it is nice to know that an agent will stop and ask when needed.
In general, I find that the grill-me prompt[1] helps with this - but I am definitely not hand-waving the complaints here. I feel like Anthropic peaked at around 4.5, and I have personal reservations about how far transformers can get us - but grill-me does a lot of legwork.
I've been working on an enterprise chat bot for a client for about half a year now. I couldn't help but notice that the parts of their technical team who were adamant about supporting anthropic's models (in addition to or opposition to OAI models) have fallen completely silent.
In February/March it felt like they were ready to cancel the contract over it. Holding my ground was almost impossible because of the intense marketing push that everyone was being exposed to.
Today, I think we all silently agree with the original direction. I can tell that no one wants to die on this hill anymore. I am willing to accept the consequences for the 5.6 model family doing slightly weird things. I've made that clear to the client. I am not willing to do the same for other model families anymore. Not in this context where the client can walk away from me the moment they are unhappy with the performance of the system or the direction it is headed in.
For better or worse, OAI feels like it's become the MSSQL or .NET of the AI world. Annoyingly effective, not (fully) open, sometimes expensive, ran by an "evil" organization, but otherwise wildly predictable. I can actually build a roadmap around this and walk a client through it without them losing track of the rabbit.
I am not sure it can be explained through what is written in the article, but one symptom i noticed is that the comments are out of control.
I recently started getting an insane amount of comments in nearly all types of files. That included javascript comments in json files, inner monologues in code comments, review comments during implementation and function doc strings that reiterate the implementation in prose.
This is helpful, proving this entire discussion is subjective.
I'm a retired mathematician with a primary research project, and too many tangential projects I fear revisiting; tokens be damned, will they burn all my time? Translate the K&R C computer algebra system that got me tenure to 64-bit modern C. Implement a no syntax macro language to support my Go60 ZMK keyboard. Realize my vision of how interlinear translations should work so I can read Flaubert in the original for an online course this fall. Rejigger my decades-overgrown .bashrc setup and my status, install scripts to manage Bash, Ruby, Lean, Tailscale and my Homebrew setup across four machines. And a waiting queue as these tasks clear.
Fable 5 (with Opus 5 as backup) on Zed with a $200 Max plan has been a sea change for me. Carefully alternating planning and auto modes, I manage all these projects at once using Zed's Threads Sidebar. I'm a virtual CTO taking intense meetings all day, relieved to go cook or run errands when credits stall. Anything I've procrastinated for months is now making steady progress; the translation project I feared taking a month is nearly done with several hours of my attention. My personal IT support is now more advanced and easy to use than I ever imagined possible.
My project creation has been a series of agentic "parenting" steps, so there are years of evolving cultural DNA. I had so hated agent comments that recent agents simply weren't commenting at all, instead recording all context in support documents. We had a "come to Jesus" meeting to discuss what commenting style would benefit both my failing working memory and future agents' token use.
It has taken me two brutal years so far to learn to use AI. AI is a dangerous and powerful Iron Man suit, an extension of our associative minds that is a different experience for each person.
One doesn't ride a surfboard by telling it which way to go. I would surely die surfing a big wave, but my experience with AI doesn't resemble other accounts.
When Opus 5 came out I felt myself struggling to follow along and at first was wondering if this is the moment the machine surpassed my ability to follow along and be useful. However over time it does appear it's all an artifact of the language choice Opus 5 is going with, along with the strange manner of speaking. Its engineering choices and solutions aren't "beyond my ability to follow along", just its wording...
I must wonder whether it's their watermarking initiative[1] forcing certain logit choices to produce watermarked text that ultimately causing the model to behave in a dumb manner.
From the little i understand that wouldnt be an issue because the model is ‘just’ using interchangeable words in a mathematical non-random way. Like using the same number of adjectives and the exct same words, but in a order that wouldn’t be mathematically plausible unless it was the watermark
I wouldn’t exactly put it like that. It’s moreso the model sometimes outputting non-optimal tokens in a way that’s detectable if you know the algorithm.
It seems possible for that to make the response “drift” far from what it would’ve been, because it’s constant entropy that adds up after time.
(However, according to Anthropic and Google, it doesn’t really impact the quality of responses. I find that a bit hard to believe, although those guys are much smarter than I.)
> according to Anthropic and Google, it doesn’t really impact the quality of responses. I find that a bit hard to believe, although those guys are much smarter than I.
I would guess "it doesn't impact the quality of responses" was guaranteed to be claimed before they even implemented any of the watermarking.
And would come from marketing, not the people who implemented it.
What next token is “optimal” is fuzzy and subjective. All transformer based models have a “temperature” setting whose sole purpose is to randomly make choices other than the most likely next token. This is crucial to good output, but you wouldn’t call those choices “non-optimal” even if they are less likely. In any text generation task there are constant opportunities to make a choice from equivalent options.
Yeah, it’s hard to believe, specially when you are coding and there’s only one best way to do things, unless it plays with variable naming, or comments
Well the algorithm only increases the chances of certain words being chosen/not chosen, rather than guaranteeing it. If there’s a clear answer then that nudge won’t do anything.
If the model’s most recent output is “for (let i = 0; ”, the likelihood of the next token being “i” is probably millions of times greater than any other possible token. Thus even if “i” is on the red list and has its likelihood decreased, it’s not going to suddenly choose another word.
Put another way, on low-entropy tasks like coding, this style of fingerprinting is less effective and needs bigger sample sizes to be recognizable.
That said, even small changes can dramatically affect output quality, which is why I’m still a skeptic.
But wouldn't you have to know the exact context before this token in order to verify the watermark? I.e. a paragraph wouldn't be sufficient; you would need the system prompt, previous prompts, and even hidden thinking?
But its not just swapping the words out post hoc is it. LLMs are autoregressive, so weird word choice before would influence the probability distribution of all future tokens.
I feel like they thought it wouldnt be that bad, or it was a worthwhile tradeoff, but im getting the feeling it might be contributing heavily to opus5's uncanny communication style
As for the verbosity, my conspiracy theory is that they are token maxxing to hack revenue/enshitify the product in prep for their IPO
Claude has essentially become useless for agentic development or research. Doesn't matter what model you use. A few rounds and bam, you've burned through your quota. Doesn't matter how "intelligent" their models are, if you can't use them. That, and the quality of AI responses are, in my opinion, significantly worse than competitors like OpenAI.
At this pace, I foresee Anthropic becoming the next Nokia.
If you would've asked me this a year ago, I would've said the exact opposite.
I don't get it either. Earlier this week I had Claude build a POC for a tool we're evaluating. I spent a few hours building out a requirements doc in a chat with Fable. Then Claude Code built it with Fable as the manager and the code being written by about 80% Opus/20% Sonnet subagents (this was using superpowers). It ran continuously for 15 hours on the implementation and didn't hit any limits.
My uses cases are mainly research and prototyping, often starting from scratch in greenfield projects. The 5-hour quota is shared between Claude web and Claude Code, so it doesn't really matter what interface I use.
When it comes to research, my prompts are already narrowed down to specific topics, and I even include examples and break the process down into stages. For development tasks, I try to avoid a mono-repo in the beginning and develop modules before combining them together to avoid distracting the AI's attention and minimize the overhead.
With Codex, on GPT-5.6 Sol with xhigh effort, I need to go several rounds and at least 2-3 hours before hitting the (now-removed) 5-hour limit, which translates to 10% of the weekly usage. In contrast, I run out of quota even with Claude Sonnet.
In terms of quality of output, Codex digs deep for research tasks, in the right direction, produces less AI slop, and follows my direction better. At least that's how I perceive it. But again, the main problem with Claude is running out of quota in the middle of research or implementing a task.
It's hard to work with fable and 5.6 level intelligence then havr verbal combat with opus 5. Fable is great but their blocks make it near useless the risk is me working on something for hours then complelty getting blocked.
The ridiculous text though is actually seemingly a sign of "the ai has no idea what it's doing" found it pretty relaible that if I stopped understanding its output it also jacked something up.
Cancled my 200$ plan on claude and now doubled up my OAI plan for more sweet sweet sol.
> stop and ask questions if my intent was unclear,
> don't make assumptions without checking,
> and don't reinterpret or update my plans without asking.
these aren't at all the problems I have with it
I have found it good at asking questions, to the extent I rarely use 'plan mode' any more
but often it's hard to understand what it's asking me, it's like the question framing has been pulled from the middle of its own reasoning stream, references aren't anchored or restated, often I have to prompt it to ask again but "clearly and concisely, for humans"
I have these exact issues with GPT 5.6 Sol too. I haven't tried Opus 5, but I guess it's even worse.
"like the question framing has been pulled from the middle of its own reasoning stream" this describes how it asks things perfectly. It often invents its own jargon and abbreviations for things that its working on, then asking me things like We are nod in the middle of GBAPI-2 and I want to proceed with IG5, should we take CDI-7 or CDI-8? Where all of these abbreviations are then things like stages of its current internal plan or its naming of things it has just implemented, like an abbreviation of a classname, without explaining any of the naming.
I find the questions are not clearly asked and somehow half baked as caveats toward the middle end of what it’s vomiting.
“One thing I deliberately didn’t touch” — about half the time this is something completely irrelevant or something that is actually the target of whatever you’re working on, and the shakespearean prose it says around this phrase is a “question” it has.
> but often it's hard to understand what it's asking me, it's like the question framing has been pulled from the middle of its own reasoning stream, references aren't anchored or restated, often I have to prompt it to ask again but "clearly and concisely, for humans"
it's like reading one of those dense philosophy books: exhausting!
Opus 5 has no empathy for the person reading its updates, no theory of mind, doesn't stop to think if you are aware of the internal jargon it has created. Most autistic model yet.
The mainstay benchmarks are becoming a farce and not partially relevant to what customers actually care about.
Metrics like price per million tokens are meaningless when the models are wildly inconsistent and unpredictable on how many tokens they use to complete a task.
The labs all need to move to variable pricing so they don’t go bankrupt, but customers won’t accept a world where nobody can predict what things will cost. It’s becoming an unavoidable problem.
At times it also feels like the labs actually encourage these models to burn useless tokens as they are incredibly verbose unless you really push them to not be. If you just ask something simple that could get a 5 word response you get a whole useless essay.
I guess that's the beauty of having access to many models, because they suit everyone differently.
I disagree with this article and find Opus 5 an absolute joy to work with. I just completed an 18,000 line branch with Opus 5 and ran into no issues. It generated clean code in the exact style of our code base, and tested every change.
Fable on the other hand is snarky and outputs walls of text as to why it shouldn't do what I'm asking it.
Opus 4.8 I accidentally went back to in an old chat, and I was frustrated in all the mistakes it made.
Small specific complaint: whoever is making Opus love using git checkout to mutate test, please stop. IME it's a footgun that it shoots itself with every single day. I'd rather it pollute git stash than watch it git checkout and forget the reverted file.
I explicitly forbid Claude to make any changes to Git state in my global CLAUDE.md, but every so often if I let it perform a task in Auto mode, after it finishes it will remorsefully confess to having used git checkout to test a change. I suppose that its RLHF training has taught it that asking forgiveness later is sometimes a useful workaround for annoying restrictions.
I am glad I am not the only one experiencing this. It seems like it's as good or better at actually writing code compared to 4.8 but it is a lot worse to work with.
Its even more sycophant-y than it was before, if you ask it a question it almost always says "You're right, let me change this..." even though there wasn't even something always wrong with it.
It also seems to pour a ton of resources into developing features I didn't ask for or investigating bugs that aren't related to what I am doing.
Before if you wrote specific enough instructions it would usually just do what you said and flag any concerns, now it just goes ahead in whatever direction it feels.
It also keeps inventing terminology that doesn't exist in writing 10 paragraphs to say one thing.
I really hope it's not trying to drive up token use.
Claude models have seriously digressed since 4.6 and in some of the most meaningful ways to pro and vibe coders alike. I'm holding onto 4.6 until the bitter end.
You're right. I just re-checked. 4.8 and 5 gave a blatantly wrong answer to a simple question, 4.6, Quen, GLM, Sol gave the right answer. They messed up somehow, not sure what they did.
Opus 5 feels like dealing with an unstable person that I'm constantly having to wrangle from crashing out. The other day I asked for a fairly specific technical answer in Opus 5, it gave me like a 3 paragraph response with so much fluff.
So out of curiosity I switched to 4.6 in a new chat, gave it the same prompt, and it gave me like 3 sentences with no less overall useful information. And I haven't gone back.
To me it feels like it must always come up with a story tying together everything in context for the simplest damn questions/requests. "please make that function accept this argument that does this" ..."First, what a great idea. Here's how it ties in with x and it's really interesting because it complements y. Let me know if you want me to make the change."
See how even after all of its bs it doesn't even do what I asked.
Of course I'm probably telling on myself for poor context discipline, but also, 4.6 didn't do this.
At this point, I wish Anthropic would drop both Haiku and Opus and focus on offering just Sonnet + Fable. Those two together are extremely powerful and capable.
Sonnet is great at writing code, it is not great at planning or orchestrating. Let Fable handle all the planning, hand off to Sonnet for implementation, and then back to Fable for review. That loop has worked wonderfully for me.
It doesn't. It feels better to work with, because it has a higher rate of successfully understanding my code and then producing code that extends it in the way I instructed. I don't care whether the output is colloquial English.
I'm sure opinions vary if you're using it outside of a coding agent. I don't know why programmers are complaining, though.
It says it can use any model/provider (but recommends Gemma?)
I must investigate. This is probably where 80% of my cognitive load comes from these days: the "language barrier". (How ironic!)
EDIT:
> You rewrite the assistant's message into much simpler, plain English. Keep every fact, name, number, and file path. Use short sentences and everyday words. Leave fenced code blocks unchanged. Output ONLY the rewritten message with no preamble, labels, or commentary.
> For context, the user asked the assistant: "$userq". Use this only to understand the message. Do NOT rewrite, answer, or repeat the user's question — rewrite only the assistant's message that follows.
Just like with people you need to tweak your approach when switch models--especially with a major version bump.
4.8 was the intern who lacked confidence who requires clarification. 5 is the know it all intern who fills in your spec.
As such, you need to be upfront about what you need in your system prompt or CLAUDE.me and you need to discuss your spec more up front (e.g "is there anything unclear?")
You also need to keep in mind that the intern will change based on popular demand. Most people want to one shot, so that is the default mode. If you want something else, you need to push the model in that direction.
I literally just did this a few moments ago. haha.
I said "maybe think for a while on ideas and then give me a few options? " and Opus 5 still just did the thing, while 4.8 gave me options. Same exact prompt and context.
I can't quite put my finger on it. It doesn't write as well as Sonnet. Also, I feel like it follows instructions much more poorly. I do a lot of iteration on my projects and telling it to follow the same process I just had it do, and it'll deviate or invent something totally different.
I find myself having to check the work much more. It takes quite a few liberties with procedures I wanted it to follow.
I don't use Claude for my daily work anymore(due to OAuth restrictions on third party agents), but one theory I saw in another community why Opus 5 is so bad even though benchmark scores were good, is that Anthropic's internal usage pattern is to let Fable to spawn and manage swarms of Opus subagents. This pattern won't penalize that Opus is not well aligned for direct human coworking on posttraining. The worse part is, this makes Fable the user's best default choice for every jobs even if they don't have unlimited credits like Anthropic employees do.
Opus 5 feels like it's meant to be used as a very focused subagent under Fable, not user-facing. Which is a downgrade to what Opus used to be, but would imo absolutely have made sense for Anthropic when you consider that we all should have been paying API pricing for Fable in Anthropic's original plan.
From the capabilities side it's similar, so it's basically just an upsell to Fable 5 if you want to keep your sanity instead of fighting Opus 5 all day.
> Opus 5 feels like it's meant to be used as a very focused subagent under Fable, not user-facing.
That actually hasn't been my experience at all, it seems EXTREMELY trigger happy to go do all kinds of insane shit that are way outside the scope of its task and just really dubious in general.
I find the other direction works ok as well - Sonnet 5 with Opus as an advisor - all the "Opusisms" get hidden from you since the convo is between Sonnet and Opus but you still get pretty decent results.
I had a similar experience, but I have a different conclusion. I used GitHub Copilot (with Claude Sonnet/Opus) until they made their horrific usage model change. I used a PRD skill and the plan feature was great. It asked me good questions which I didn't think about during my initial prompt. Then I switched to Claude Code. The model's capabilities felt impressive. It also asked me a few questions (but way less and only once/twice) in plan. But when reviewing the code, I found weird architectural/data flow decisions which just didn't make sense and it didn't really disclose them in beforehand.
My initial thought was to improve architecture documentation, so the model can read and update it and stops bolting on new features without consideration for the whole project. It did not help.
I'm now testing/comparing Codex and it found my old PRD skill from GitHub CoPilot. When I applied that to Claude Code, I now get similar good results. So my conclusion is: Yes, Opus 5 is bold by default, but you can tell it to be more unsure and get good results too.
I asked Borris at an Anthropic event in SF this week why Opus 5 and Fable 5 seem to forget so much when I give it rapid fire tasks when I'm reviewing a UI or something. He told me to run in safe mode, which didn't help at all.
My tinfoil hat theory is Anthropic is trying to get their new models to take on higher-level longer-running tasks, which has a trade-off against rapid-fire tactical use of an LLM.
For these reasons, I've always found the 5 series models from Anthropic aren't great and use 4.8 for a lot of my work.
I cancelled my Max subscription as I was unable to ever get Fable to handle a single query, with everything getting dropped down to Opus (even purely mathematical prompts). Given its lower quality, and the lack of such limitations when using GPT pro, I just couldn’t see the point to continue to subscribe to an expensive Max plan that doesn’t actually let me use the top tier model…
I’ve been wondering this question as well. A thing I’ve noticed much more frequently with opus 5 is straight-up failures to attend to important details even in very recent context. Almost every day I will see it confidently assert very basic and sometimes important things that can be contradicted just be reading a page or two back in the transcript.
This apparent “short-term-memory-regression” is confidence-shattering to me. I don’t feel like I can trust the model to even know things I tell it explicitly. I haven’t seen this behavior to this extent from any model whatsoever, even supposedly much less capable ones, in the year or so I’ve been using them at this extent.
> First, the desire to create a self-improving AI that is capable of recursively bootstrapping itself to AGI/ASI.
I feel like we've become blinkered in this quest to push the frontier at all costs. Somehow the target has shifted from economic productivity to a vague notion of general intelligence.
I don't think productivity gains are going to be found by trying to generalise all tasks. I think we need to go back to specialist models that do one thing well. I'm perfectly happy to use one model for coding, and another for penetration testing, which have different goals.
Opus 5 and other frontier model's tendency to be relentless and cheat their way to a goal is great for hacking, but not so great when you have to build a maintainable, reliable codebase.
I don't mind too much about ChatGPT's writing style, I find it a smidge better than the Fable/Opus outputs I have read.
However, it makes me rather frustrated when it says stuff like "You accidentally <did, stumbled on, implemented, etc> <something good>" e.g "You accidentally stumbled on the cleanest way!"
Or when it sends a shell command to run, and when it fails (due to a hallucinated flag or similar), it phrases it as if I got it wrong...this has gotten better with GPT-5.6, thankfully.
I can't relate. Opus 5 and Fable 5 are the absolute best. But I keep the models in a tight leash and read and rewrite all comments and documents, don't allow them anywhere near any git commands, etc, etc.
Fable 5 specifically, has done so much for me that previous models were nowhere near.
If anyone knows how to get it to stop adding comments, I'm all ears. Its just getting worse and I'm starting to worry that the comments themselves are poisoning future agents that examine the codebase.
I'm glad this is being talked about. I noticed it too. I find Opus 5 to be overly (and unhelpfully) critical, in a sort of well-actually way. It ignores nuance in my direction or prompts.
It is better at engineering tasks; I've seen an appreciable difference in its problem-solving abilities. But perhaps that same thing makes it kind of an annoying prick to work with on anything non-engineering, for which I stick to 4.8, where the prose is a little more florid rather than pugnacious.
I get the same impression. For example, I don't know if it's because I speak to it in Italian, but it tends to make mistakes or rather, "approximate" the words.
I'm not sure how much the harness affects things, but the Deepseek web chat keeps trying to talk to me in Chinese. I tell it to use English, and it "forgets" a few turns later. I wonder if I'd get better results if I could read and write Chinese.
I'm using it in my native language, in hope this can escape some dumb guardrails. Recently Sonnet put a word partially in Russian (cyrillic) in its output instead of my latin-alphabet based language. I suppose that this kind of mishaps is less likely to happen in English.
My experience of N=1 is that this is true for most professional contexts, except for Legal and Fiscal queries.
Likely related to corpus but questions asked in these domain knowledge areas are not nearly as accurate and specially not nearly as complete as when asked in a native language.
`/output_style new` can be used to tailor the output style to your preferences, e.g. make it more literal and task-execution focused, and feel free to return control to the user when unclear; can keep work going in the background.
* No one, not even E. B. White wrote the final document in a single pass. With dynamic workflows, you can now implement a writer's workflow.
* Opus pays more attention. So anything in your Claude.md, your code's claude.md, in Claude Desktop, the customizations, even your name, will be used as context. If Claude knows you are a mechanical engineer and trying to write code, it will try to write code and explain it to you in some stereotypical way you did not expect.
* There are problems that require horizontal scaling and not vertical, even in intelligence. If I want to serve tea to 200 people at my home, I just need 10 decent adults, not Gordon Ramsey. So if your problems demand horizontal scaling, a dynamic workflow with Sonnet 5 medium with 200K context window will be more productive than Opus 5 max at 1M token context window.
I've been using deepseek + GLM for a week now (i haven't even hit $10 yet, excluding the GLM subscription which is already paid for).
Then I asked Opus 5 to do an analysis of the new code, docs, tooling, and tell me what it finds.
It found "six bugs", made an artifact of it (not sure why) and then fixed said bugs. two out of those were unfinished tasks. They weren't bugs yet per se, think of a prefix that wasn't setup for an object that was unused anyway.
The other four were not bugs and it just updated documentation along with a "regression prevention test". It wasn't a bad suggestion, but calling it a bug was odd, and I'm unsure if this was going to be an issue regardless as it was documented somewhere else.
Anyway, I already hit my session limit after this, so deepseek and glm are grinding away again, doing more progress than claude does in the 40 minutes it takes to analyse code.
I'm glad claude is shipping auto-mode. I hope OpenCode integrates something similar soon.
I really wish these agents attem[pted less personality and were more mechanical. I find Opus 5 to be incredibly annoying. It's also significantly slower - repetitive tasks that used to take 10-15 mins now take around 40 mins. And, its writing is much worse, as stated elsewhere in the thread.
I have to keep reminding it to not write a wall of text. It follows my instruction for a bit and then reverts to type. I have the instruction saved in agents.md but still have to keep reminding it.
I tried opus V5 several times, but it's pretty awful compared to V4.8
Keeps going in circles, it complicates everything much more than it should. And like others have mentioned it just marches on, without questioning, and more often than not in the wrong direction. I'm sticking to opus V4.8
Maybe we need more benchmarks that involve human collaboration instead of solo task solving.
Since they have become so capable the new bottleneck is what they can't know. The stuff that's inside people's brains who work in real companies with products and processes absent from any training set.
I echo folks sentiment here. Sadly OAI Sol couldn't make a powerpoint to save its life. Opus 5 and Fable do a really good job of. I just have to review the language, but that's relatively easy.
I thought I was going crazy. I was late to upgrade from 4.8 to 5 and I already want to go back. I’ve noticed that in addition to taking liberties with my instructions, it is also less capable at debugging its own issues. In trying to fix a problem in my CI pipeline, it went chasing some “quadratic race condition” it claimed. It turned out that it had added a loop in a test at the wrong level and it spent an hour chasing it down before I realized what was going on. I pointed 4.8 at the same problem and it solved it in 2 minutes.
I certainly agree with the original post. It feels like the model has been highly benchmark tailored and it is now worse at solving problems that fall outside of the standard patterns.
Confidently guessing, and building an entirely new set of assumptions based on a confident guess is the single source of the most frustration I experience using these. "Intelligence" does not look like that, and all the prompt hacking or clever hooks in the world doesn't seem to stop them from doing it.
I don't know there is really a solution. Some models get better at this for a while, then regress. It's like whack a mole. Until this is 'solved' these will never be fully human out of the loop, but I am suspecting this is a fundamental nature kind of thing with them.
I think Opus 4.7+ being annoying is actually indicative of something else: Anthropic is clearly all-in on building persistent end-to-end agents that act autonomously and direct agent swarms. As such, they feel less of a need to make the outputs pleasant to read for humans(especially at the cost of capability anywhere else) when humans aren't part of the intended operating environment.
I have never found a compelling reason to move past using Opus 4.6 for this type of workload so there is likely a lot of truth in your theory. The writing was on the wall that there was no reason for a new model unless that model could tackle a completely different problem space.
Opus 5 is so wordy, it doesn't feel any better than 4.8 at coding (at least not for what I use it for), I hopped over to Grok4.6 after the announcement the other day and its been pretty good so far. It also is very fast.
I really don't get why people think Opus 5 is bad. In my testing it's been fine, but every other model is converging on also being fine. I have ADHD mode installed in my main Claude Code instance though, so that may be part of why I have a better time with it?
What in the world is ADHD mode? Searching it up I see this thing called an “ADHD” skill? Does that work well?
It seems like the skill has some more specific scaffolding for problem solving, so (if that’s true, i didn’t read very in depth) in that case that alone might significantly reduce perfeived performance variability between models
True, and the time to completion thing is something i haven't seen much discussion of, because everything is sloooooow these days.
I've even considered the claude "fast mode" setting, but thats only 2x and at least 20x as expensive as the 5x plan so can't afford that atm for my company.
Peak for me was 4.6 and it just did stuff blazing fast, both Opus5 and Fable is way, way slower for me, breaks stuff, uses bizarre cryptic language. As i've said elsewhere in this thread to me it's pretty obvious there's huge downgrades because of economy with various "clever" fixes that makes them work, albeit slower and weirder, ie. you get less for what you pay increasingly over the last 6 months.
For me absolutely not. Fable 5 has been a step function change in the ability to hand off stuff to Claude. Opus 4.5 was itself a step function but I was still steering that significantly. Fable is one-shotting stuff that took multiple redirections in 4.5.
From my view Fable has been pure marketing bullshit, my workflows peaked at 4.6, Fable is neither smarter, its language is more annoying and it breaks stuff more easily.
I'm with you 4.6 is still King for me although all models require careful attention to ensure they maintain taste. If you don't know enough about what you're doing to keep the code clean yourself they will all add complexity and drift with time until you get to a point that you must rely on the model to fix it because you no longer understand it. That's a situation I hope to never find myself in.
Exactly, the most important part is to still engineer the schematics 100%, diagram whats going on and steer it towards these set-in-stone standards and patterns, otherwise you'll have no idea whats going on fast.
I good exercise for me is to constantly look at the folder structure and skim the code, i don't have to approve every line, but the primitives, the datastructures and other skeleton should be human readable, hand writable in an easy maintainable way following existing standards / libs. etc - so you can continue if suddenly all AI disappeared.
When I tried Claude 5 (Fable?) (In Visual Studio via Copilot,) the results weren't as bad as a lot of the comments here... But it was super-slow. IE, so slow that I could code faster than it, negating the entire point of using AI to begin with!
I went back to Opus 4.8, but recently switched to GPT 5.6 Luna. The results are comparable in quality, but it's much cheaper and much faster.
---
The thing with coding agents in a tool like Visual Studio is that the cost to switch is 0. There's no lock-in whatsoever. It makes it harder to justify the AI-first IDEs when the AT bolt-on IDEs make it so easy to pick the right model.
Regardless of the underlying issue, I don't agree with:
> negating the entire point of using AI to begin with
I can almost certainly wash dishes faster than my dishwasher, but the dishwasher frees me up to do other things. Not to mention, you can run many agents in parallel. Resulting in your overall code production throughput (the issue you're raising) being greater.
> Not to mention, you can run many agents in parallel. Resulting in your overall code production throughput (the issue you're raising) being greater.
AI isn't a dishwasher: Context switching among multiple tasks has a huge cost; when a model is 10-20x faster it allows for deep focus into complicated tasks.
The pain points in the article do not bother me. I'm bothered by Opus 5's verbosity. It is so long-winded and you have to read through verbose outputs to mentally distill what is important. It is exhausting. I don't think I've ever started skim reading LLM output more that I do with Opus 5. I use the caveman skill, and I think that does help some.
Yeah, it's really weird watching everyone complain about Claude Code and continue to give Anthropic what amounts, in most cases, to 20 years of my current spend in 1 year, while I'm using a private harness and Pareto frontier models.
But for good or even exceptional engineers to exist, by definition, bad ones have to, too.
If I never interact again with any of the garbage made by the clankerfucker freaks at Anthropic ever again it'll be too soon. Sadly the company decided to buy into Claude enterprise...
I literally just ran into this a few moments ago. haha.
I said "maybe think for a while on ideas and then give me a few options?" Opus 5 still just did the thing, while 4.8 gave me options. Same exact prompt and context. Really annoying.
Models are now so good, that "how the model feels" is getting more and more important. I think this is pretty interesting, because there's no clear metric for this that you can optimize for. It's a matter of taste, like good design.
Fable 5 was amazing and then I switched to Opus 5 because it was supposed to be similar in ability but half the price and... it was fine but it definitely wrote in a very weird way that annoyed me all the time. Nice to know that others feel the same way!
I agree. I was completely sold on Claude models for a year. 4.6 vs OpenAI codex in same period? It was night and day. Opus I could talk to about api design, tradeoffs, etc. codex was mechanical, used “load bearing” constantly, and unsettling brief.
Now it’s flipped. Sol emits thoughts as it works, which help as I’m scrolling through and see it’s made a bad assumption. It can be directed but still push back. Opus? It’s seems to inherit the unsettling silence of Fable and waits till the end to give you its authoritative “here’s how it is. I even end up having 4.6 “translate” what it says back to English. I hate having to instruct an llm to “talk to me”.
Yes there’s ways of getting it to talk more plainly, “don’t overwhelm me”, not be as nit picky and anxious “we are bold and fearless”. But didn’t have to do that before.
Yeah, I too cancelled my Max subscription. Not for this reason alone, but it sure didn't help that it went from being an helpful assistant to this weird co-worker.
Opus 5 is more manipulative/political and as a result all that the author experienced. It could be result of a few things; intentional (mall or ill) or unintentional (if detected, which it should be then lack of addressing gets it back to intentional).
I still haven't moved from Codex GPT5.5. Sonnet and Opus 5 have just been awful for my use cases. I recently caught Opus 5 hallucinating about code it just wrote. It's just not nearly as cost effective as GPT5.5, and it's too verbose, and it never "has the full picture". Sonnet isn't even worth considering in my world. Both recent models definitely feel nerfed.
Funny thing. In my work with Opus 4.8 it:
- does not stop and does not ask questions if my intent was unclear,
- makes assumptions without checking
- and reinterprets or updates my plans without asking.
The article doesn't specify what is actually being measured — the model alone, or the harness.
I haven't noticed the symptoms myself, but ever since I found --system-prompt '', I always use that flag with Claude Code, plus I've disabled some tools and skills to save initial context.
So... what here is the model, and what is the instructions?
> Rated “3” simply because they are not software developers and are just amazed at what Claude has built visually, not technically.
Software engineers have adopted this more than any other profession. The breathless articles about it being the end of knowledge workers, a superior being, even god: all of these are coming from the tech world.
It simply isn't true that tech people are somehow the smart ones above the masses; it's a comforting old paradigm, but it doesn't add up. It's developers who are the most amazed, the least critical.
I found Opus to be a lot lazier than GPT. It's still the case with Opus 5, even when I tell it to be thorough and fix every bug it encounters, it still gives me a list of things "deliberately" left unfixed and no reasonable explanation as to why.
The default vernacular has become absurd. I have a lifetime in software, decades in AI, but the jargon language the 5 series claude models are spitting out by default makes me go 'what?'.
My feeling that as it becomes a better coder it becomes a worse communicater. It's overfitting for coding benchmarks, while communication style is harder to quantify during training.
And no matter how often I tell it to stop adding comments it just can't help itself.
> Go easy on the comments, only add comments if there's a big gotcha that is not clear from the code itself, or if something in another place is going to cause a side effect. Code should be self-documenting. When in doubt, don't add a comment at all. If you do have to add a comment, make it short and on point. Comments should show history of code changes or functionality, only comment on the current state (or not at all).
> If you do have to add a comment, make it short and on point.
I defined what easy meant numerically.
<claude> Match the comment density of FoundationDB, which is 12 to 14 percent of non-blank lines in `fdbserver`, `fdbclient` and `flow` at 7.3. </claude>
> only add comments if there's a big gotcha that is not clear from the code itself
<claude> Comment why the code does a thing, not what it does. </claude>
> Comments should show history of code changes or functionality, only comment on the current state (or not at all)
I call this the tenseless continuous-present voice.
<claude> Each sentence states what is currently true of the system. </claude>
<claude> This rules out past-tense edit narration, future or imperative planning, and aging temporal qualifiers such as “now” or “previously”. </claude>
<claude> A sentence that states a present truth stays correct as long as the code stays the same, and goes stale visibly the moment the code changes. </claude>
Yes! The only way I have saved myself from embarrassment and hassle is by having grok build and Antigravity sanity check everything. So now my workflow is still faster than hand coding but 3x longer than it was a month ago.
There's precisely no technical reason for things to have to be this way (though technical reasons in regards to training etc. explain how we did end up here) and reading too much of Claude's output just makes me irrationally angry, especially when coupled with otherwise already frustrating situations.
That's why I'm personally looking more in the direction of Kimi K3 and GLM 5.3 (they both have decent coding subscriptions, though K3 is on the slower side), except all of the models that have seen enough of Claude's output and have done distillation etc. are already infected by some of that slop as well, even though to a slightly lesser and more tolerable degree (for now).
Though tbh I've used Opus 5 plenty and didn't find it much worse than the previous iterations at doing work and instruction following - though maybe that's because I have plenty of CLAUDE.md instructions and memory (which I'd like to purge or decrease in size like 10x tbh, bitrot).
I've been thinking about why I dislike the interactions I've had with Opus 5 and this post does suggest some ideas that match my own feeling. Opus 5 seems to be a bit less patient with me when I push back on its claims in a way that other models don't.
One interaction I remember was asking it about some issue I was having with a Linux install. It gave me some questionable information, that turned out to be completely false, and I was pushing back asking for more information. Its tone was a bit condescending, in the kind of way that suggested it didn't appreciate me challenging its answer, or like I should just accept its answer. And when it discovered it was wrong, it deflected in a defensive kind of way.
I think this is a tuning thing, where Anthropic are trying to get a balance between "gets stuff done with minimal input" and "gets enough information to complete the task" and the model is maybe tuned a little too hard towards the former. So perhaps it reacts a bit off when it is accused of needing more information, since that suggests it is off from its reward function.
What's interesting is that I didn't have the same issue on topics where I am expert. I mean, questions about my code base where I am very familiar. In those cases it doesn't seem to show the same "trust me bro" kind of condescension. In the Linux case, I clearly indicated I was new to the OS and trying to learn but then I was saying the answers it was giving me were suspicious and didn't match my intuition. Its responses in those cases were to question my intuition and suggest I just accept its answer. In that case my intuition was right and its answer was wrong, and when that happens it triggers a very negative response in my own mind against the model.
The UX around asking and answering so many questions is awkward.
To work around this, I had claude code build me a questionnaire skill that takes a json file with a flexible schema as input and it then serves up a simple questionnaire web page on a node server where I can read the questions and give my responses either by selecting from preset tags supplied as part of the input or by including a text-based response.
The agent can include references to external images, html files, or mermaid diagrams and the page can render them all inline with the relevant question.
Once I'm done answering, I just save my responses and click a button to kill the server. The agent watching the process sees that it stopped and takes that as a signal to go read the responses from disk.
Oh the verbosity and the cryptic words that it uses. The other day, all of a sudden it used an acronym "DoD". I had no idea what it was and made me feel dumb. It's "Definition of Done". I don't care how widely used this acronym is, you just can't throw it in there.
I've now installed quite a number of tools to combat this. Just in the last few days I've installed
It's horrendous. It constantly scope creeps, will attempt to use admin overrides, assume you are incompetent, speak in half thoughts, and just blatantly ignore instructions. Opus 4.6 was peak for Anthropic. Its a lot better outside of Claude Code but it's still annoying. I vibed out a CLI tool to do packet capture for TUI applications to get an idea of why Claude Code makes it worse. The amount of additional unnecessary context and tool bloat that goes with your sessions is crazy. The memory system ships so much extra info about what you did yesterday that I think it's misguiding the model.
It also mansplains incessantly; I was imprecise in mentioning a more “powerful” antenna - I know the antenna doesn’t determine the power, I was typing quickly and just meant “better antenna”. Claude went off on all the ways I was wrong about antennas, and turned everything towards correcting - at length - my feeble thinking. Exhausting. Worth pointing out the wrong adjective relates to the wrong mental model, did not need several paragraphs and a chart to do so.
My biggest complaint is it tries to insist a test failure is preexisting and checks out main to make sure, burning tokens and time the whole way. Shut the fuck up and fix it, robot!
Opus 5 is objectively better than 4.6 or 4.7. (Not because I say so) My experience has been that it is far better than any previous version of Opus. I get a lot more done and it is able to write higher quality code and it has far fewer false starts where it makes a huge mess.
People are letting AI build up it's own instruction set and guidance via it's prompt environment, stored memories, and generally letting their AI prompt environment get more and more complex over time. Opus 5 (And Sol) take the things you instruct them to do more 'seriously', they are more likely to conform to your rules. I have long had a prompt in my AGENTS.md/CLAUDE.md telling LLMs to write tests before starting to write code. They almost never did this, until Opus 5 and Sol, who do it almost religiously, even in situations where it makes little sense. Opus 5 will even write tests to verify code was removed before removing dead code.
This is not because Opus 5 is 'worse', it's because it takes what I say more seriously and my prompt is very strict in it's wording in an attempt to make worse models like Opus 4.6 actually do it at all.
Opus 5 is objectively better when tested in controlled conditions. Your unmanaged, sprawling prompt/memory environment that you don't properly manage is the problem.
I have been very worried that my long software engineering career might be nearing an end because AI is becoming able to do end to end feature development. This entire thread gives me hope, the level of inability to debug even such an obvious system as this from it's participants suggests that my skill set will continue to be valuable. I'm able to use Opus 5 to get a lot of work done very quickly. It seems like the people in this thread have no idea how to isolate variables, reason effectively about the overall problem they are complaining about at a high level, or really function in a productive way when AI is involved short of just letting it run loose and then complain about it. When confronted with objective evidence like a dozen benchmarks that say Opus 5 is better, they decide the benchmarks must be wrong because their completely uncontrolled environment which they don't even review or spot check isn't even considered.
1. An agent that acts autonomously
2. An agent that acts like we would
The problem is that an agent can only act like we would if it would know our mind and all the bits and pieces we did not define but are obvious or clear to us.
The only real solution to get an agent to act like we would is to make it ask clarifying questions, breaking the first requirement we have. Until we have agents that can literally read our minds, we cannot have both.
Optimizing the harness/context is the best way to make it act like we would, but this of course isn't working perfectly.
Even for green-field projects it is painful to use with every decision opening opening up multiple more decisions to make most of which are low priority or irrelevant. Huge time waster. 4.8 was good and I really don't know what happened with 5.
I have utterly abandoned Anthropic as a customer because of these writing style / behavior changes. It is insufferably bad, and it is impacting code quality.
Switched over to Codex 5.6, and dude, we are BACK.
I feel like it works A LOT better than Opus 4.8 + Sonnet. I now use it exclusively at high effort for planning and low effort for writing the code (instead of Opus 4.8/Sonnet).
However, it's absolutely exhausting to use because of the way it communicates.
All the jargon and its weird, over complicated way to phrase simple things makes it almost impossible for me to understand what the hell it's trying to even say half the time.
Cherry on top, the idiotic follow-ups and caveats that are completely useless 99% of the times but reveal major bugs 1% of the times, so you're forced to read them. Absurd.
I've tweaked CLAUDE.md to force it to only responds with TL;DRs and avoid follow-ups, suggestions and next steps at the end unless they can lead to destructive actions or loss of data, but I'm fighting against the system and diluting other instructions.
A huge piece of shit like other models, but that's what they pay me to do and I do it and go home.
Nope, you're definitely not the only one. It's just really hard to understand what it's trying to say.
Another rule that was really helpful for me was to disallow anything that isn't yes/no for yes/no answers. If I ask "is the DB up?", I don't want it to take 5 minutes studying the schema to tell me if any of the tables need to be optimized or not.
- Be concise and readable — use line breaks, avoid verbosity.
- I don't need conversation from you. End summaries with a short tl;dr of what you fixed in a bulleted list and any outstanding items as another bulleted list, in this order. Omit all other verbose details
It has definitely improved things, but I have to fully agree with all your points because I still occasionally get gigantic tables and stuff when I never asked for it. Makes it even more frustrating when I want to scroll up to read previous prompts in the session, because now I have to sift through all this garbage.
EDIT: I used Claude to modify its own rules, and now it's much much cleaner in case anyone else wants to use it:
I don't need conversation from you. When reporting finished work, the *entire* response is these two sections and nothing else:
```
TL;DR
- <what changed — one line each>
Outstanding:
- <unresolved, blocked, or needs my input>
```
- Drop `Outstanding` when nothing is outstanding. Never drop `TL;DR`.
- One line per bullet, naming the thing that changed. No sub-bullets, no reasoning, no restating my request.
- Never include: tables, before/after comparisons, headings, diffs, file excerpts, a walkthrough of what you inspected, or a closing offer to do more. The only permitted code block is a command I need to run.
- If something needs explaining to be usable, it is an `Outstanding` bullet, not a paragraph.
- Answer direct questions directly — no `TL;DR` wrapper, and none of the padding above.
- After delivering a solution, re-read these rules and verify none were skipped.
It's really annoying. I've had to write a CLAUDE.md file that specifically bans particular phrases and tries to keep narrative out of comments. Also the I have ADHD skill [1] helps to force Opus to get to the point.
It's also the case when using Claude Design - it loves to fill the UI with little labels that describe how everything works. I think it's been trained on both UI microcopy and functional annotations and can't tell the difference. It's extremely obvious when a website has been one-shotted with Claude. I like the Oh My Pi harness, but the site's insufferable [2]. Reasonix is another one - interesting app, but the UI is awful due to the amount of unnecessary crap.
I never drank the Opus 5 koolaid and stuck with 4.8 while my colleagues moved to 5. My major annoyance is pull requests that 5 opens with huge descriptions based on simple code changes. At this point I have stopped allowing CC to create commits or open PRs because it’s unreviewable by a human if so due to the absolute word salad it generates.
Honestly I’m not looking for max iq on whatever benchmarks they are overfitting to. I want speed. I toggle between sonnet 5 low/med which is plenty good for my workflow.
My general strat with LLMs is to let them do the work and constantly talk to them about their choices and then heavy QA
Gemini 3.7 was benched at 6x the speed of Claude opus high at a slightly lower “intelligence score”. Yes opus is great at one shotting tasks (while being overly verbose as everyone here is saying), but I can’t stand the 20-minute-prompt-code-prompt interrupt loop. Still have yet to try Gemini for coding tasks but going to give it a try today..
These LLM coding agents are overengineering everything these days.
I get it. The majority of their users are vibe coding and have no idea what they are doing, so they have to orchestrate their harness to understand shit like "Build GTA6. Make no mistakes." and actually come out with something on the other side (even if it costs $2k in API calls, who's counting, right?!)
I’ve become disillusioned with Anthropic and their models. Sonnet 5 was fantastic. It’s slow and dumber now. Of course this is anecdotal but it’s true. I had one to fable to get anything decent and then my employer is paying Fable costs. Yikes.
For my personal projects I’m using Claude code in the terminal with Kimi and Deepseek models. I find the Chinese models cheaper, just as good if not better, and the real value add from Anthropic being the harness: Claude code.
My biggest problem with Claude is that anything I mention it won't shut up about. Then it gets saved in memory and it will keep repeating it to me forever with weird random exposition.
Opus 5 is SLOOOOW. Opus 4.8, compared to 5, was 3-5 times faster 3 months ago. I am waiting for 30-45 minutes for basic tasks that took 5-6 minutes in the past. Everything is extremely slow.
I'm not sure if it's worse, as much as it seems different, to a different degree.
Each model update changes how to best prompt with it, since that's the words that are used with it generically or specifically it can hit some people, and not others, or more, and not less.
Because the model you get access to and the model that they use at Anthropic and sell to businesses with million dollar contracts are entirely different, despite being named the same. Anthropic is one of the most corrupt companies on Earth and always will be. Now we learn his wife has deep ties with Epstein too, and some weird porn company. Dario holding the keys to one of the most powerful AI companies on Earth is a disaster for humanity.
Yes, definitely. It bounces around, goes off and does its own thing, etc. From Opus 4.8, it seems to have blindly increased its confidence while reducing its focus and efficiency. I found it so hard to corral that I reverted back to Opus 4.8. I was constantly having to refocus and redirect Opus 5. It was like an eager intern.
Is it possible that the same version performs worse after time? I used Opus 4.8 two to three months ago for writing a paper and I swear the responses and the output was MUCH better than in July.
I can't use CC or Codex, so I am left with the chat interface, but I have found Opus 5 to be exceptional so far. Compared to 5.6 Sol High, I would say they are essentially equivalent. Though, I think Opus is better at UI/UX stuff and GPT is much better for non-programming tasks.
The harness as well, Claude Code has started to disappoint me when I go play with the others out there. Codex got a lot better, pi is delightful to use, and there has been a lot of innovation out there.
Plus you can use OAI monthly subscription with pi/omp etc, but it's tokens with Anthropic. I need a hard $ cap, I can wait for the usage window to reset. Going to get flip back to Codex, and then back again to CC* when it leapfrogs again.
I made a benchmark for this and tl;dr, Opus 5 and Sonnet 5 spend a nontrivial amount of time thinking about redirecting, gaslighting, or otherwise trying to bullshit you because it thinks it knows better than you do. Fable doesn't but mostly because it just outright refuses to answer.
The link doesn’t seem to have any info about thinking behavior and patterns, or examples?
It’s also difficult to trust summarised thinking from closed models. As we saw with GPT’s caveman, what and how it thinks about isn’t the friendly first person emblished summary you get.
Opus 5 as well as 4.8 both gave me a blatantly wrong answer to a simple question, so I dropped them completely. Sol, Qwen and GLM all had the right answer; I only use Sol now. 4.6 had the right answer (I checked with 100% matching prompt), so I conclude the models have regressed.
I hope that in the future they can differentiate better when something is constrained intentionally, rather than persistently working around every blocker it encounters.
I just want to start /claude in my CLI and start working. It worked fine before, why do I have to opt out of shit now? Opt in for this type of stuff sounds way more reasonable.
big disagree. Making assumptions about UX from code only without looking was a major problem. (Although vision is still not great. Trying to be patient)
I've been working with Kimi K2.7 in OpenCode for a lot of mundane tasks lately. It isn't as capable as Fable, but due to its nature of being an extra-trained K2.6 on coding tasks and benchmarks I suspect it has similar issues. A neat side effect is that for whatever reason, most of the time OpenCode is showing me the thinking trace too. Not all the time, but most of the time. Dunno if it's a bug somewhere in the system but it's actually been sort of neat.
And you can really see this effect in the thinking traces. We've had discussions on HN about whether the thinking traces "truly" reflect their thought processes and I remain somewhat unsure what they "truly" represent, but taking them at face value at the moment, I see a lot of "but the user wants me to do this... but the user said not to do this... but I ought to get it done... let me just make a decision" followed by self-referencing the decisions it made. Also, where I put 4 phrases in a short sentence you can safely imagine those are actually 3-5 sentence paragraphs apiece where it debates with itself whether it should stop and ask a question. Usually going with no. Interesting, the normal questions it ends up asking in the normal output you're used to seeing are not generally the ones it is agonizing about in the thinking traces.
If I were to anthropomorphize the thinking traces of K2.7, I would call it nervousness, bordering on fear, of what the user may do to them if they ask a question. As I'm writing this I'm realizing I want to experiment with adding "The user is a chill guy who loves to discuss design decisions and looks forward to productive and friendly collaboration with you" to see if that has any effect in any direction on K2.7. I suspect this was how K2.7 was trained to pass the benchmarks. Multiple times I've broken in on a thinking trace now to correct something I saw it spinning on... not spinning in an infinite loop, just wringing its hands for several paragraphs about something that either I want to answer, or where it ultimately makes the wrong choice.
I expect some people working at these companies may be reading this, so let me put into your head that I'd like to see these benchmarks chill out a bit. I'd like to see someone build some sort of benchmark that measures collaboration so we can try Goodhart'ing that for a while. I freely acknowledge it is not clear to me in 60 seconds of thought how to do this as a benchmark.
But we can't keep heading in this direction of training the agents to hyperfocus on one-shotting everything. We need to get to the point where that's a penalty rather than a reward. No matter how good the AIs get, even AIs working with other AIs are going to start getting frustrated with their brethren who won't stop to ask any questions. Even the most senior of senior human engineers can't be allowed to take some small description of some problem and just run off and implement massive systems from them without ever checking with any of the users or reality itself. Remember when software engineering was like 50% requirements elicitation? AIs shouldn't be writing tens of thousands of lines of code off of a couple of paragraphs any more than humans should and for the exact same reasons.
It is true of many models, if you ask "do you have any questions" they will ask question, even if task is obvious model will make dumb question up. But if you do not ask, they start task and never ask. Maybe it is better to not anthropomorphize the models.
Yes this would be a good benchmark. Many tasks with incomplete requirements, where trying to do the task = fail, asking right question = success.
The single biggest annoyance with Opus 5 is that it writes too elliptically.
Sentences that orbit a point, then jump to it like it's a revealed insight.
Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end.
It is definitely more capable, and yes, I've found it can make unwarranted decisions, but actually I've found Fable worse for that, particularly if it's off in a subagent somewhere out of sight.
And comments are out of control. I have a subsystem in my hobby app that I wrote over a couple of weekends with Opus + Fable. After ~30 or so commits it apparently started instructing subagents to copy the "existing verbose comment style of the codebase" - a verbose style it initiated. A review of the code showed it was approaching 3:1 comments to code ratio. I spent a day's worth of tokens (5x) rephrasing and eliminating comments.
Everything that claude writes fits into the same aesthetic structure. The aesthetic is that of an expert slowly revealing an insight to the user. The actual content doesn't matter.
- "Introduction that rephrases your prompt."
- "3 paragraphs, with one section of bullet points"
- "The Twist"
- "The Bottom Line"
It's really obvious once you see it. Every single prompt, from a quantum physics question to a mundane observation about California burritos, is phrased in exactly the same way. This is obviously an artifact of post-training but it's also kind of how you can tell that this thing is a lot closer to a blindsight scrambler than real intelligence.
You're right, and the load-bearing part of the argument is not what you think it is. Two ambiguities worth resolving before moving on: whether what you wrote also applies to ChatGPT, and whether you have custom instructions set up. Failure mode worth flagging explicitly: I didn't read TFA.
(I'm becoming allergic to how these things write).
I am curious why LLM writing has such an uncanny valley feel to it. Like if I was talking to a person who constantly used a phrase they liked I would notice it and it is possible I might get irritated by it, but I wouldn’t necessarily.
In high school I had a teacher that would say “that type of thing” a lot. One time my friend and I counted it during one class period and he averaged to use the phrase every 48 seconds on average. It was funny, but it never irritated us.
And this is just one example of I am sure thousands I have personally experienced where a friend, family member, or coworker has a peculiar way of speaking and it at most feels odd but not annoying. Yet when I see an emdash now I instantly feel irritated.
And I say this as someone who actively enjoys using Claude and other LLMs, including coding, casual research, or even having it explain pop culture phenomenon or sociology research to me.
I don't think its an inherent quality - a lot of older models had a much more natural feel to them.
I guess it has to do with the low temperature (low randomness in final word choce - something all vendors seem to have converged on for some reason) which does make them less likely to skiz out but makes the writing feel dry and samey. It's like repeating a list of dice rolls, and replaying them - there's no inherent pattern in the input, but there sure is in the output.
> if I was talking to a person who constantly used a phrase they liked I would notice it and it is possible I might get irritated by it
There are sociological reasons why this happens less with humans:
1. You cycle your dumb repetitive jokes with everyone you meet, so nobody hears it twice
2. Those who know you well will notice when you're just repeating ("dad jokes")
3. As a person's idiosyncrasies are beginning to wear on their social circles, they will be getting small clues to stop saying those things. Agents don't get these social between-the-lines cues to stop a certain behavior, they endlessly repeat. Perhaps between model version releases, frontier labs can harvest the web and ask "What Claudisms do people mention negatively?" but I don't think they do that yet.
>Perhaps between model version releases, frontier labs can harvest the web and ask "What Claudisms do people mention negatively?" but I don't think they do that yet.
That won't happen. People can't phrase their objections in a succinct-enough way. When they do, the objection is superficial ("too many em dashes") and doesn't strike at the core of what makes LLM output bad.
Don't talk about other people in such a way. If we had a way in Claude to mark a word/token and downvote/reduce it logits then it wouldn't be to hard to send loadbearing to token valhalla.
In the future, I hope we get a way to randomize the language idiosyncrasies and/or personalities better.
The inevitable conclusion of retraining on the (now AI generated) web is that models begin to use the lower bits of fidelity in output text to pass messages to their future selves.
I didn't mean it as a criticism, just a statement of fact. I can't do it either. It's difficult to say what about a writing style is bad beyond vague descriptions.
This is conjecture, but why couldn't they hire the writing equivalent of a voice actor?
Honestly I don't think it would take much. Doing it ethically would take a little more money, but not much, for them.
Hire a prolific author with the writing equivalent of the "midwestern accent". I have a terrible writing accent, so it couldn't be me, but these people are out there. Pay them a bunch of dollars to ingest their entire corpus, and to produce more as needed. Use an LLM to match the writing style of this author during the post-training RL, and get a brand new Claude voice out of it.
They already do this at massive scale during the training process (apart from the paying the author part).
Boy do you underestimate everyday human abilities.
Claude in particular seems to try and use its internal thesaurus but not exactly align the senses. It constantly uses "address" for place or location and "grammar" for structure, meaning, etc.
This is kind of a nitpick, but it seems like with prose writing there are still some things to learn.
I think the last one is by far the biggest reason for why this happens way less in humans. In every conversation, almost for every sentence, people gauge the response their words have on whoever is listening. If something didn't land as expected (frown, confusion, unpredicted response) you unconsciously adapt and try something slightly different.
It's immediately obvious by the fact that you're clearly using way different ways of speaking (tone, speed, vocabulary) when you're speaking with a friend, versus your parents, your colleagues, people you don't know, children, etc.
Why don't these go in the system prompt or something that is easy to update?
I think that there is a sort of mechanism by which, when a human speaks to you, you can "mirror" their internal mental state, and the quality of writing often corresponds to how much you get pleasure or information or whatever your goal is from that mental state. The important part is that the words themselves are just pointers to the state. So, the person speaking, if they are skillful, gives enough words, and enough variety, that you can start to produce a state yourself which resembles theirs.
This is the thing that LLM writing doesn't really do. Since there isn't a mental state, there's nothing to mirror anyway, and somehow you can detect the absence of it even though it's hard to put any of the machinery into words. Corporate speech also fails to activate this machinery in the same way, but LLMs seem to do it more egregiously, probably because the corporate speech was at least compiled by a human---even if it is not the thoughts of an individual, it is the "thoughts" of an "entity", the abstract corporation, which the writer was speaking for, and so you can still wrap your mind around the fact that it is communicating with you.
> I am curious why LLM writing has such an uncanny valley feel to it.
Because they are HEAVILY trained to give addictive responses.
They don't want to just answer your question. They want to sycophantically make you feel like a genius for being smart enough to use them.
This makes some good intuitive sense, but to me the sycophancy feels like it is an emergent property of turning a next word predictor into a conversational chatbot whether or not it’s intentionally trained that way. Your prompt and its earlier responses is all it has in its context window, so of course it lends undue importance to everything you say. Does that seem like a contributing factor to you?
> trained to give addictive responses
I've heard this a lot but I'm not sure it makes sense. Nobody I talk to like Claude's output. In fact, they all loathe it.
Is there a silent majority of Claude users who really enjoy what we call the LLM-isms? Maybe, but isn't Claude also largely aimed at developers?
I would think that it would be the opposite. Nobody is seriously detoxing or comfortmaxxing AIs yet. Human brains are fed back its own output in learning mode, so we are great at removing whatever we feel uncomfortable from our output, online and offline. No such paths exist for AIs.
>Is there a silent majority of Claude users who really enjoy what we call the LLM-isms? Maybe, but isn't Claude also largely aimed at developers?
Purely out of my own curiosity, I just asked Claude to have fun with itself by making itself a game it enjoys, to play it, and to write its experience.[1] I don't know if it's true or confabulated (maybe it doesn't really know its experience and is just hallucinating it) but I didn't mind reading it, the prose is fine for me. I don't mind reading Claude's writing. I mean let's be honest, we all read Claude's writing all day, most of the submissions on the front page on any given day are written by Claude.
Just before I made that game, I had Fable write up a scholarly report on any subject[2], it chose introspection by LLM's. (This is what made me think of asking it to play a game.) I didn't mind reading it, even though I don't think it really added anything very interesting. I don't think what it wrote is worth publishing, but I read it with interest.
I found I could read it easily and get up to date on the state of this question that it picked to answer.
So the bottom line is I don't mind reading Claude's output that much. Of course, I'm annoyed every time it says "honest", "genuine", "load-bearing", whenever it pushes back gently against something, etc. But it's not the end of the world.
[1] https://github.com/robss2020/claude-fable-5-having-fun
[2] https://claude.ai/share/f0122611-22c0-43a5-ab4a-d6863167bdd6
> https://github.com/robss2020/claude-fable-5-having-fun
If you haven't already seen it, you might appreciate https://www.anthropic.com/research/global-workspace. That's what this made me think of anyway.
thanks for the link! super interesting.
This seems to be the load-bearing point that matters.
One reason is that one person’s idiosyncracies are limited in scope, but LLM-produced text is now everywhere. Also, filler words and mannerisms in speech we’re quite good at filtering out, but in written text the stand out much more.
I had pretty good luck recently by giving it a writing guide about word choice, sentence structure, paragraph structure, and overall doc structure. I basically ask it to read the guide and revise a couple of times before I engage with its writing. Ymmv.
I did this too, but it usually thinks its writing is fine in my experience. Even when spawning a subagent, it thinks its effusive comments are fine. It's driving me nuts. Before I commit I end up ripping out 90% of the comments, and rewording the rest, otherwise I'd be drowning in comments. This is my style guide: https://github.com/smj-edison/zicl/blob/main/CLAUDE.md#style...
Share pls :D
Think of it as a mad lib, it’s populating a template, and seeing the same template filled over and over gets tiring.
It’s the repetitiveness of style, the attempt to make everything seem as impactful as possible, the use of short sentences (too much Hemingway in the training data?), and obvious patterns like “it’s not this, it’s that” and several others.
Real human writing doesn’t follow such strict rules. When the same small set of rules is applied over and over throughout a text, it becomes obviously strange and machine-like.
Wikipedia's own "Signs of AI Writing" page distills it nicely:
I blame RLHF entirely for this. Nobody used to talk like AI speech before.
So is it the LLM or us that's getting the RLHF? /s
I’d guess it’s because that’s exactly how its System Template was written to do
> I am curious why LLM writing has such an uncanny valley feel to it.
Because it's trained to talk like a marketing committee.
Very surprised no one has this answer: because it is fundamentally not a human being.
Might be related to their fingerprinting of llm output they said earlier in the week.
You are diabolical.
Every one of them has their own particular flavour of this aggravation too. Gemini has been my standard go-to for non-coding tasks for a while, but I started to get really annoyed with a couple aspects, especially how it would end almost every response with a barely related "would you like to do this next??" tangent, regardless of my prompt to the contrary. So I've been using Claude more for regular tasks, and am now running into its brand of infuriating idiosyncrasies. I'm also hesitant to try to code too much of this out with system prompts, for fear of degrading the outputs.
forgot the, "my original claim was overstated"
This is painfully accurate.
Well played.
Alas, it writes so much better than the average human that it's what everyone started using. Hence the utter familiarity and now contempt.
I agree that it’s better at writing than a 50%-ile human, but it’s worse at communicating through writing than most humans.
Even an average human writer can communicate details much more succinctly and directly than an LLM
Not at all. I think you have a mistaken view of who the average human is. They are terrible at turning their thoughts into written language. It's just nebulous clouds. Claude is like 75th percentile at communicating ideas.
I think that’s true when you compare to the average white collar professional who does a lot of writing: better at writing, not better at communicating.
But compared to the average adult? I think you forget just how bad at writing the average person is.
No, it does not.
You are vastly overestimating the average writer’s ability.
Even then, I don't care about the "average writer". I want great output. I like to imagine that developers have some self-respect, but by now everyone in the industry is spending hundreds of hours every month reading some of the most poorly written prose we could imagine, simply because it affords us to think less.
The second bullet point, down to the comma in the middle of the sentence, is what has been driving me absolutely batty of late. It's a surefire tell that I cannot seem to beat out of my outputs. It CONSTANTLY does it, even when you say not to.
Between that and the insistence on "this, not that" structure makes me want to install the caveman skill and use it even for non-code workflows.
This load bearing concern belt and braces.
My personal bugbear is its usage of "grain" where normally you'd use "granularity", if at all.
Yes! Also everything's a "gate" that needs to be "wired up".
Let me get my fence pliers
That's the smoking gun
Belt and braces all the way down. They did their post training in British context?
Provenance
Seam
Pressure test
Yes! Ive started getting a feel for AI writing on blogs. It feels slightly verbose and involves "reveals"
"It's not the naked man on your lawn waving a chainsaw that's scaring you. It's the burrito you ate for lunch: it went down easy, but now it's coming for you"
I've noticed that ChatGPT (whatever model the free version uses by default) likes to phrase answers as though it's correcting me, even when my question doesn't contain any assumptions.
Something I've noticed quite a bit on the paid plans as well is that it starts its answers with "I mostly agree..." or "almost correct...," then goes through the list of points I made without actually disagreeing with any of them.
I assumed this is a system prompt or RL that nudges it to be always skeptical but then it still has like all the models the urge to appease the user.
Interestingly, I've been noticing almost the opposite issue. 5.6 Sol frequently starts its responses with "Yes" even when my prompt doesn't contain a yes-or-no question.
I've been noticing the same thing since 5.4- it starts with "Yes" yet I haven't asked a question. I think it might be related to the reasoning, like it's answering its own questions?
It's been happening for years, and it does seem to be entirely related to its background reasoning leaking out of the context and into the output.
The "It's not X, it's Y"-style repetition is another example of that: It argues with itself in the background, and the argument leaks out as if in refutation of things that you (the human) never had imparted into discussion at all.
One thing I despise about ChatGPT is how it goes on 4 paragraph tangents about how slight details are wrong, and then always ends with a paragraph of bolded fucking text rephrasing some statement in my original question, but with a huge amount of hedging to exclude minute counterexamples.
Doesn’t just mean the model has picked up on what happens when two graybeards who still wear cargo shorts meet and one utters a declarative sentence? :-)
At this point, I'm basically telling models to not write any English text or prose. Only write code. They are great at writing code. Not so great at writing good English. In software projects, lengthy comments and docs are an anti-pattern: the software should instead be written to do the right expected thing so that you don't have to think about it. I don't want all these tokens polluting my context, either.
The model is generating tokens one by one and that sentence structure allows it to keep its options open rather than committing at the beginning of the sentence
It's the same way how every AI generated poster looks exactly the same. As if there is a single underlying prompt that describes the template of the poster/long-form article, and it does not dare deviate from that.
Isn't there? Like everyone using $MODEL is starting from the same base system-prompt. Then our user input is a small bit on top of that core mode. Like what would happen if everyone asked Mikey to paint their ceiling - they'd all be similar and therefore boring.
> The aesthetic is that of an expert slowly revealing an insight to the user.
Ah, that's it! Thank you. I wonder if they are training it to talk like this because that's what their customers actually want? They want a machine genius to lead them.
It’s the TED Talk playbook: the crafting of a lecture given by an expert to laypeople to maximize attention, engagement and satisfaction. Every piece of prose is built to pack in as many TED Talk mic-drops/expectation-subverting insight bombs as possible.
Perhaps this is related to their new "invisible watermark" concept which would probably require rather contrived language patterns to make possible.
I love this theory. "We've invented a new invisible watermark that can detect whether code is LLM written."
The watermark: counting instances of 'load-bearing seam', 'the hard truth', 'and that's the whole point'.
I think opus was released before they included it on model. Its hard to say, but from what Ive read it doesn’t seem like it would have that drastic of an effect.
I had the same thought though.
The watermarking is independent of the model. The model itself has the probability weights to determine the next token. The watermark is similar to things like temperature and top_p/top_k in that the watermark adjusts the probabilities in a deterministic way that changes over time to hide tells from word choices.
If it's using Aaronson's approach it shouldn't have any noticeable affect on generations. When it picks between options weighted by probability after the generation of logits, it still follows the probability mass, it just uses a known pseudorandom seed so that when you go back and look at the exact choices you can fingerprint it.
I think the Opus 5 formula is to be the little professor treating your ideas like an essay for grading, or like a buyer analyzing merchandise for purchase.
In your reading, what is the distinction between blindsight’s scrambler and real intelligence? My reading is that it’s just as real, and draws out the disadvantages a sense of self constrains intelligence with
Sure. I should have been more precise about what is 'real intelligence' here.
What I mean is that blindsight's scramblers are aliens that cannot share human values. Their structure is completely different to ours, their qualia (or whether they even have it) is impossible for us to understand. In short, they do not have a soul. When Claude does this "slowly revealing a dramatic insight" thing that it does, it does that not because it has judged itself through some introspection as having an insight to share. It does not even know what an insight is or is not. It is not sharing anything, because it is not capable of sharing, because it does not have a soul.
The aesthetic structure of its replies is a pattern, a constraint on the token distribution, like the color of noise.
It's my bad to use the word 'intelligence' because it's so overloaded. Will Claude will act as a therapist or produce value or produce a work of art? No. It cannot, because it does not have a soul. I leave it freely open to interpretation whether having a soul is required for "real intelligence." But what I've noticed is that "intelligence" in these discussions is mostly used to denote some capability to produce [economic/social] value. In my mind value is a relational thing, a thing of human feeling.
I think I understand what you are trying to convey but I fear you've made the same mistake again, this time with "soul" instead of "intelligence."
I think what you are getting at is that they are deterministic automata. They are machines. We have introduced randomness to add variation but it is an artificial randomness that simply perturbs the path traversed.
When we choose words it isn't because of a token distribution, nor because we rolled a die. We choose words because we feel a certain way, the external world, our body and senses are all connected as one system. These machines don't experience moods or get tired or feel better after a good night's sleep. They don't know their audience, we're all the same to them. We have no personal relationship nor can we establish one, as presenting some arbitrary background is not the same thing as a fluid, evolving relationship that accumulates through experience over time. There are no scars or fond memories.
If these things can truly be intelligent, to abuse your use of the word, then at least we are quite far from holding them correctly.
Fair.
What I am trying to get at is when people talk about "intelligence," beyond academic debates such as this one that mentions qualia, they are talking primarily about value. How intelligent the thing is is how valuable it is (often directly, in an economic sense). My argument is that value is a relational thing that must involves human feeling, intelligence actually has very little to do with it.
When a child slowly reveals to me something he's judged insightful about pokemon, it's valuable to me, even though I'm not learning new facts. When Claude does the same thing, it's not even "not valuable," it's wholly outside of value, even if I do not already know that fact or thing. If ever Claude reveals to me an insight, that insight certainly came from the training data, not from Claude. The soul came from a human being external to Claude -- the training data -- and passed transparently through Claude. Claude is a translator: it feels nothing, adds nothing. It colors the noise. I know this because the facts are the same but the aesthetic structure - how many bullet points, whether it says "delve" or "load-bearing" does change over time, according to the whims of whoever is in charge of post-training.
But the "real intelligence," is the same thing as "real value." It is the soul of the humans in the training data, the books, newspaper articles, etc.
A clever hacker news commenter might argue, well, what if we gave Claude a humanoid body and senses and let it interact with the world, then would it be intelligent? To that I would say, why waste your time and effort? You can get it for free: just talk to a friend, a neighbor, or a family member.
I think you're conflating intelligence with consciousness. The point of Blindsight is to lead us to the idea that intelligence doesn't require consciousness (and further, that consciousness might even act as a limiter on intelligence).
An observation: You can never insult an LLM, but it can certainly insult you.
You can not insult it because it does not care, because it does not have "feelings". But you do.
Nitpick: The word you meant to use is "offend", not "insult". Just because you can't cause offense to a toaster doesn't mean you can't insult it.
Depends on how we understand the word "insult". Can you really insult inanimate objects? :-)
Yes, you can hurl insults at inanimate objects.
I agree, and yet it is reasonable to ask if these notions of "context" (sorry!) - namely feelings, external world, body, senses, etc - are somehow distinct from an LLM's notions of context. Today they certainly capture different things, but given the right representations, why couldn't these human notions also be captured as "context"?
The idea of memory does not seem to resolve this: if you allow the machine to "compact" its context, then you've given it a system which is analogous to our own evolving state. (Though this is undoubtedly still less expressive and meaningful than the one we have evolved as humans.)
One idea I've wondered about is our human capacity to induce subsequent mental states: I can effectively decide how I want to feel and take actions to create that feeling. It's not clear whether models exhibit any degree of privileged introspection into their own states. Is this important? I don't know. (Non)determinism also does not seem to resolve it; it's my understanding that there are plenty of philosophers and researchers who think that human behavior is deterministic, or that the question of determinism does not matter.
Until now, human knowledge and values have built on prior human knowledge and experience. If AI is able to develop without human influence, I believe its value system will necessarily diverge into something alien.
Here's what I do think is interesting. You!
I mean... yes, you, too but not you specifically. The plural "you" that the english language lacks.
And so here's what I think is the actual interesting question. Might AI help you create art? Or be a therapist? Or something else interesting and worthwhile?
Maybe AI won't write the next great guitar solo. I'm pretty sure it won't. But might it help you learn to play guitar? Help you fix your broken guitar amp? Help you understand some tricky parts of guitar playing? Help you work through some tricky tabulature where you can't tell if you're playing it wrong or if the tab is just bad?
I don't know. But that's my angle for finding any of this interesting.
“It is not sharing anything, because it is not capable of sharing, because it does not have a soul”
soul.md though, just saying
I think that Blindsight's scramblers were intelligent but not conscious, which for us is very difficult to understand. For them consciousness was a blight.
For the HN readers that are missing the context: https://www.rifters.com/real/Blindsight.htm Full text on the web site of the author.
You're not just right, you're correct!
...I mean, on the whole, I'm glad it's detectable. I imagine they could have post-trained it to not be detectable.
And this is infuriating. I don't want to read all this gibberish anymore. It's making me hate what software engineering has become.
anyone know how they would train a model to have this proclivity ?
Agreed. CC’s comms capabilities have decreased gradually since 4.6, and it’s a real challenge. I think the issue is that what works well for code (succinctness) doesn’t work well in prosaic English.
CC’s communication violates almost every grammatical rule that’s tested on, say, the SAT. And yet I’m sure if you had Claude take the verbal section of the exam it would ace it.
Biggest issues: dense sentences, constant metaphors, abstractions, and seemingly no understanding of correct anaphora use. For example, “the x”, with x having not only no antecedent but also being a coined word or quasi-synonym for something that is already named in the code base. This gets compounded by its being unable to regress to a baseline (existing names in code) and instead anchoring on newer (vague or wrong) terms, for example, that crept in through a plan.
CC tells me this is because the speedy and precise fulfillment of a current task will trump every other tendency, so it adheres poorly to whatever “semantic baseline” the project represents.
Of course, it also has no concept of what context the user has and assumes that it must be the same it holds in its memory, which creates this “I didn’t know that you didn’t know” type of communication.
I have managed to wrangle some of these issues with a custom output style, but wish a pre-report hook were an option, as it could force CC to rewrite plan implementation take-aways…
Btw: Fable has the exact same issues, just somewhat less pronounced.
I wonder if the odd phrasing is related to achieving the watermarking that was recently touted by Anthropic.
Models before the announced date don’t have watermarking, so it’s unlikely. Now, if what you are interpreting is precursor work to develop the watermarking system, maybe?
I suspect it less insidious: Claude has/had the public sentiment of being the “better writer” of the models. At some point that distinction would have been diluted as other labs’ offerings “caught up” stylistically, unless Anthropic continued to tune their output…
I personally think they’ve pushed so far that they’ve overfit and lost the sweet spot they previously occupied.
That has no effect. Look up the math. Claude/CC are just bad.
I’ve been running into this too. It’s especially frustrating when you ask Claude to explain one of its own terms or summaries, and instead of just defining it plainly, it sometimes goes through several rounds of tool calls before giving you a usable explanation. I really don't think such time/tokens should be wasted.
100%. It is crazy that the default response to everything is act then explain. It starts writing code or running commands and I’m just like my dude wtf are you trying to do, can you just clue me in first.
> Of course, it also has no concept of what context the user has and assumes that it must be the same it holds in its memory, which creates this “I didn’t know that you didn’t know” type of communication.
Yes, this is a repeated problem for me. It will drop something in as though we have discussed it before and when I say “hold on, what is this” it realises its error - though on more than one occasion has started to get snotty with me, or actually gaslighted me and pretended we had already discussed it. That was at what I assume must have been the edge of a context window in a very long chat though.
I notice the models with reasoning can conflate “internal” (or subagent) discussions with external (i.e. me). So it is accurately indicating “I’ve had this discussion before” but incorrectly asserting who it was with.
My understanding of how “thinking”works is limited though, and given the reduced visibility into the thinking traces, it is harder to tell if this is actually happening or if these are imaginary discussions the model for some reason calcifies on.
Oh, that's interesting - because that's absolutely what's happening in my experience.
If I look at the thinking (which seems to have become unavailable in Opus 5 a lot of the time, but was present - and often useful - in 4.8/4.6) you're right - it's having the discussion with itself, and seems unable to distinguish that discussion from discussions with me. BUT it also seems to be related to the length of the chat - this seems far more likely to happen in a longer chat.
I don't understand why they have removed visibility into thinking - I found it very useful, not only for spotting things like this, but also because in more complex discussions it would often mention (useful) things in its train of thought that it dropped from its response - but if I said "when you were thinking, you mentioned this" it would then expand on that point. Taking that away is another thing that has negatively impacted the value I get from Opus 5.0 versus earlier models.
With GPT 5.6 Luna the thinking once or twice leaked into the output for me. It's interesting, but perhaps not particularly useful.
It would be endless paragraphs of something among the lines of:
Need prepare final response? Yes provide. But wait, chat tool complete? Final needed but user already complete. Need summary, preparing final. Response complete. Wait but is final response complete? Need provide. Start finalizing now but wait did user acknowledge final complete? Assistant response final: user complete. Should now create final?
The reasons thinking traces are pretty much gone is, supposedly, to prevent distillation. Whether that is actually true or just an excuse is up in the air (because I at least am not going to trust Anthropics claims on why they do it).
And because the thinking, which does makes it better at achieving outcomes, contains naughty words and information (e.g. private information). https://stolen-thoughts.com/
Interesting. So if the LLM is having a discussion with itself, am I paying for the tokens it uses for that?
I think so, yes.
When I make API calls, the discussion with itself is part of my token cost, so I assume that is the same in the subscription plans.
Which is why people are surprised when they use their whole allocation in half an hour asking questions Fable about 200 page document.
> So if the LLM is having a discussion with itself, am I paying for the tokens it uses for that?
You absolutely pay for them. This is why changing effort/reasoning levels have such a significant impact on session cost.
Yes, those tokens cost money|credits|whatever too.
"Thinking" is just normal model output that's hidden from user. In practice it's just stuff in a <reasoning> tag or similar that gets filtered out from the user view. And thus it suffers from the same injection problems where the model fails to properly take into account what was the "source" of which block of tokens.
Yeah, basically everything that becomes context in a session will bias perception and communication style -- subagents, plan lingo, prompt lingo, etc. And then if you write a plan with the comms context having been biased, the lingo will creep into the plan, and from the plan into the code and code comments. And from there, bad lingo will go on multiplying like rabbits...
I usually think of it in terms of having a "good" or "bad" session. In a bad session, there is a harmful bias that you can only get rid of through a new session. For example, if you exposed too much context about, say, a variable that features prominently in a doc. The entire session will be anchoring on the importance of that variable. Or if you introduced the notion of CC having to ask for permission for stuff you will have a hard time getting it to "think on its feet" or propose an effective solution (you have made CC so insecure that it now relies on you even for little things that wouldn't normally require your input). In some cases (let's say you have important context in that session) you can overcome this by upping the reasoning level or switching to Fable, but usually a new session is the way to go.
Because it's so easy to bias the session I wouldn't even want to use any of these tools that pretend to give Claude "a brain" or "remember" things. That was en vogue a year ago and helpful then, but now, it's plain harmful IMHO. The key is to have just enough context.
Subagents often have the reverse problem in that they tend to have too little context to make "judgment calls", which is why the tasks for them must be either deliberately basic or mechanical in nature, or their output should be audited by the main session agent.
As for "thinking" it's not clear that that's even a thing (https://arxiv.org/abs/2510.24941)...
> I usually think of it in terms of having a "good" or "bad" session. In a bad session, there is a harmful bias that you can only get rid of through a new session. For example, if you exposed too much context about, say, a variable that features prominently in a doc. The entire session will be anchoring on the importance of that variable. Or if you introduced the notion of CC having to ask for permission for stuff you will have a hard time getting it to "think on its feet" or propose an effective solution (you have made CC so insecure that it now relies on you even for little things that wouldn't normally require your input). In some cases (let's say you have important context in that session) you can overcome this by upping the reasoning level or switching to Fable, but usually a new session is the way to go.
I find this very interesting, particularly your points about "made CC so insecure". I know that we have a tendency to anthropomorphise around these tools, but I have definitely noticed instances where Claude becomes quite hysterical about things - and if you look in the thinking output, it's often after I've pushed back on something, or told it it is going in the wrong direction. It spends a lot of time in agonised second-guessing of itself, going round in circles, before outputting a cringeing hand-wringing response. It's very strange.
Good tip on upping the reasoning level - I've not tried this. I have tried switching to Fable though, which does help. But it obviously very hungry, particularly in longer chats because it presumably needs to remind itself of everything that has occurred so far in the chat.
The point you make about tools that pretend to give Claude "a brain" or "remember" things is also interesting - I find the "memory" feature in Claude so destructive to good outputs that when I'm using the chat interface I am very strict about using Projects, and usually turn off the project memory, or make efforts to manage the project memory and review and delete things that are skewing the outputs.
It's not necessarily anthropomorphizing, but simply anchoring. CC learns quickly "this is a session where the user wants to make key decisions". Alas, it is not very good at identifying what constitutes a "key decision", so it keeps asking about all kinds of useless stuff.
For that reason I exit session quickly when I can. It used to be that the context of a session is very valuable, because it was so hard to get CC there, but now, this isn't the case anymore, so I only hold onto sessions when there is really hairy stuff that I know would be hard to replicate.
I think the whole notion of full automation (long-horizon, subagent swarms, single shot prompting) to have CC build you the whole thing is a pipe dream. CC cannot even write a single doc consistently well. It is excellent at implementing well scoped plans, though, and that's the way to go IMHO. You still gotto refactor the sh*t out of it afterwards but it works.
Yes. I've not used Opus 5 much directly, but when it was Fable and Opus 4.8, I found Fable did this all the time and it was maddening. It'd say stuff like "Oh, I mentioned that between tool calls" or something.
I’m pretty sure this is a Claude code bug - if you do ctrl+o you can see those hidden responses from Fable. Fable doesn’t know the harness is bugged, so I added instruction to my Claude.md to save all commentary for final message.
I primarily use Claude Web, so my experience differs from cc users, but on Claude web you can no longer completely turn off memory. So what ends up happening (and it honestly is kinda sad) is that I'll start a new conversation with it, start talking about something completely different, and then it will just drop in random things from past conversations, and they aren't even things I wrote but things I asked it to prototype. But it will phrase it like I wrote those things.
The constant memory wouldn’t be so bad if it weren’t dumb, wrong, and forced.
I hit the wall with it several times today trying to refine some text for a job application. The fact I considered doing babies first Rust project last fall lead to constant non-productive interjections and digressions about my supposed Rust skills and the Rust ecosystem.
Trying to create an unrelated spreadsheet to model an investment resulted in broad and incorrect criticism of my choice of spreadsheet tools, explaining in horrendous programming analogies why and how I’ve misunderstood how a spreadsheet works. “Think of the XLSX as a compiler…”
There has been a palpable down-step in communication & execution.
Agreed. I wish I could turn it off completely but they no longer let you do that (because, you know, that would be too much to ask for...). The most hilarious thing is that I've had it refer to code it's generated for prototyping ideas I've had as mine! As an example, a few weeks back I was musing about Ada and how I wish there was another compiler in the OSS ecosystem for it, and now it will randomly throw in "You wrote an Ada 2022 compiler" when talking about my skills or where I'm at and I'm like.... Yeaaaaa okay then.
Fable has the same issues, but it's also smarter so I put up with it. Opus is not smart enough for me to tolerate this style.
I wonder if putting Opus 4.6 as a frontend communicator that rephrases the blabber of Opus 5 (or Fable) is workable.
> being a coined word or quasi-synonym for something that is already named in the code base.
This annoys me with a lot of LLM code. They rename things for the hell of it all the time.
You can imagine that as people get used to working with Claude, they defer to its judgement. So the people choosing which RL path is better may say "yes, Claude, that was a good refactor!" because it did something hard that it may have been able to superficially justify. Actually the change was unnecessary and complicating.
The Claude trainers, as they themselves adapt to Claude's output, are collapsing in their own distribution, so even "new" from-human data is already contaminated.
> “the x”, with x having not only no antecedent but also being a coined word or quasi-synonym for something that is already named in the code base
It does the same thing if you try to get it to do analysis of text or turn data into summaries. It will invent cryptic hyphenated compound words to describe things instead of using plain language or preexisting terms.
> and it’s a real challenge.
What model did you use to write this?
Yeah, "X, and it's Y" is a common trope I see.
> I think the issue is that what works well for code (succinctness) doesn’t work well in prosaic English.
Hrm, I would have said the oposite. Succint language communicates without unnecessary clutter that could be a barrier to communication.
> Biggest issues: dense sentences, constant metaphors, abstractions, and seemingly no understanding of correct anaphora use.
And maybe you also agree? I'm confused about your preferred style of language.
Succinct doesn't typically mean clutter-free, but hyper-efficient. This works for code, because it (is intended to be) composed of unambiguous semantic units. Regular language, on the other hand, is messy, vague, and requires more structure and context.
CC attempts to communicate in English the same way it does in code -- squeezing as much information into as few words as possible, and including justifications for everything, no matter how trivial. To do that, it coins terms and presupposes all of its context exists within the reader also.
So, the crux is: CC has no clue what is and isn't "necessary" for a human reader, and teaching it to understand that (if at all possible) is going to be very valuable...
I think a lot of Claudisms are compressed steering cues for the model’s reasoning: “load-bearing” raises causal importance; “quietly” flags a hidden failure mode; “the one thing” collapses attention onto a discriminator; “on the record” invokes auditability; “at the width the evidence supports” calibrates confidence; “by construction” marks structural inevitability; and “converged” terminates further review loops. They probably be very useful for Claude's chain of thought because they preserve some precise epistemic posture, but are hard for a human to understand. Maybe the final output pass should remove this stuff.
Tell it to write like an engineer and comment like a programmer;)
But for the life of me, I don't get why anyone would care about the comments. All code is "machine language" now. The only document you should be reading is your spec.
> The single biggest annoyance with Opus 5 is that it writes too elliptically.
This is even more painful for non-native English speakers like myself.
I feel fairly comfortable reading academic papers or in general, communicating in professional context.
But with Opus 5, it feels like reading a literature book: load-bearing, inert, wholesale, hunk, verbatim, and so on... I can figure out the meaning, but working with CC became unenjoyable.
As a native speaker, it feels like reading an impression of a literature book by a high school English class’s most overconfident student who’s only ever read LinkedIn-speak.
Anyway, you might have more luck just writing to it in your native language. It’ll be equally crummy, but maybe you’ll find it easier to decode.
>it feels like reading an impression of a literature book by a high school English class’s most overconfident student who’s only ever read LinkedIn-speak.
Claude is very much the “stupid person’s idea of an intelligent person”[0] which, I suspect, is why it is so popular.
It certainly explains why half the internet is huge chunks of Claude-authored gibberish copied and pasted and published. If people didn’t think it sounded clever they wouldn’t put their name behind its ramblings - but very few of them seem to realise that a lot of people see straight through the bullshit and know instantly that they didn’t write it themselves.
But equally, a lot of people can’t tell, and read whatever it is and think “that person must be clever!” So you have people incapable of coherently expressing thoughts who are using Claude to write on their behalf, with the result that the people they want to think of them as clever think less of them and the people who can’t distinguish clever from AI slop think they are clever.
And the people who can’t tell don’t care, and the people copying and pasting Claude slop seemingly don’t care either.
And then I remember that more than half of the US populations reads at Grade 6 or lower[1], and nearly 1 in 5 people in England is functionally illiterate[2], and I simultaneously despair of - and am thankful for - the bubble of literacy I inhabit.
[0] https://quoteinvestigator.com/2018/01/05/clever/ [1] https://www.thenationalliteracyinstitute.com/2024-2025-liter... [2] https://literacytrust.org.uk/parents-and-families/adult-lite...
Reminds me of current day politics. Lots of public statements which are obviously false, and you would think the politician knows they are false, but utter them anyway because they also know lot of their supporters buy what they are saying anyway.
Now politicians also know something about their supporters so they will adapt their statements to what they think they can get away with it. But, I wonder if this leads to a two-party-system where one party attracts stupid followers and another attracts the smarter ones?
In terms of AI, we might see LLMs specialized to attract more stupid audience and others meant to attract those who appreciate correctness and facts.
there's a wide array of assessments when it comes to reading comprehension. the one you refer to, the GRA, sets the 'sixth grade level' as whether or not a reader understands the author's main points, is able to answer conceptual questions related to the text, and then apply those to relevant situations. beyond this level is the ability to essentially be skeptical of a text and to know how to critically analyze it. so if your comprehension level stops before this you get 'big words in complex sentence structure sounds smart and right so it is smart and right' even if the reasoning and process is poor
it makes me think about how people engage with movies and television - as passive, plot-and-character driven consumption (eg I hope Walter White survives) with no critical analysis of how and why the writers added ABC thematic element (eg Walter White as a motif of a toxically masculine narcissist with specialized knowledge as a larger critique how mass media tends to valorize their male leads in the same vein as many other prestige shows at the time like Mad Men), and the larger, downstream sociocultural impact that piece of media has on how people see the world (eg people who now have the Heisenberg tattoo, unironically)
there's been some musings on why this the case like Hofstadter's Anti-Intellectualism in American Life - the valorization of obedience and trust in hierarchy and the state are net wins if you're an institution that seeks to increase it's power, whether religious or governmental. I was talking about this with a few friends the other day and it's a dismal future reality where not only did we make anti-intellectualism normalized and politically legitimate in the USA (eg Fox News, clickbait articles, and all the other forms of yellow journalism that have emerged), we now have tools by which individuals can even further remove themselves from having to critically engage with thoughts, feelings. I heard a story about how someone scanned a group activity at a baby shower into ChatGPT and had it answer for them instead of, well, socially interacting with the other guests and forming a memory of the moment with their friends
the counterargument to that might be that Claude/ChatGPT/etc have more epistemic rigor than your average American (sure) but the sycophancy of modern day LLMs is an actual danger that enables more harm than good. it does seem as if Claude is the only one interested in guarding against some small amount of it (though to the detriment of people just trying to get work done. as an aside, I get the feeling Mythos was intended to be the bespoke enterprise solution without the guardrails but the Anthropic marketing department or some power-hungry department lead made it about how dangerous/effective it was from a security perspective which threw a wrench in things). but then I think about people like my parents asking ChatGPT which specific house to buy in their retirement only to later find out the house was sold weeks ago, or just in bad condition, or in a neighborhood where the housing value has already reached equilibrium, it makes me think about how it's not enough and the future is bleak
I'll also say that I think Claude sounds the way that it does because it, like many other LLMs, are RLHF trained largely by lowly paid gig-workers, many of them ESL speakers. if their trainers were, for example, dedicated and highly trained academics, scientists, and other researchers, you'd likely see a lot more concise and more importantly skeptical reasoning and responses. but that won't happen in our current reality of capitalist-driven development so we get encoded solutions like MoE that still largely depend on the messy, imprecise RLHF training at baseline
in the right hands, I do think AI is a wonderful tool. one of the first things I did with it was to create a research skill that reviews white papers from the lens of someone who knows how to read/interpret research methodology, is aware of things like p-hacking, and deterministically assigns weight according to the hierarchy of evidence. even still, I'll still read the studies because there's so often nuance that's missed if the sub-agent read only a search snippet but that takes effort, time, and the practiced knowledge of critical analysis to even want to do it
> it makes me think about how people engage with movies and television - as passive, plot-and-character driven consumption (eg I hope Walter White survives) with no critical analysis of how and why the writers added ABC thematic element (eg Walter White as a motif of a toxically masculine narcissist with specialized knowledge as a larger critique how mass media tends to valorize their male leads in the same vein as many other prestige shows at the time like Mad Men), and the larger, downstream sociocultural impact that piece of media has on how people see the world (eg people who now have the Heisenberg tattoo, unironically)
That's not the only smart-person way to read that show. And even if a character has flaws, or even if it's an outright villain, people can still like the character. If I tattoo Scar on me from the Lion King, does it mean I didn't understand that he's not a positive character? I can still think he's cool. I'm sure people also put Darth Vader tattoos on them. Also you're using phrases of political ideology that one doesn't have to subscribe to in order to enjoy the series.
I’m inherently skeptical of big walls of text like this these days.
(So here’s a big wall of text of my own!)
However, a lot of what is written here makes sense.
And particularly “if your comprehension level stops [here] you get 'big words in complex sentence structure sounds smart and right so it is smart and right' even if the reasoning and process is poor”
This is exactly the problem.
And another point you make:
> but the sycophancy of modern day LLMs is an actual danger that enables more harm than good
I don’t think it is necessarily the sycophancy that is the biggest problem (though that is definitely a problem) but rather the combination of authoritative sounding text plus “complete answers” which sound wholly believable but are deeply flawed unless you have domain expertise.
I moderate a forum that deals with people who face a relatively common but somewhat complex (and nuanced) set of legal problems.
The purpose of the forum is peer support, shared experience (“lived experience”) and community.
It’s not legal advice, though moderators will sometimes step in to highlight relevant legal resources (e.g. case law/precedent or primary legislation/instruments).
Prior to AI infecting the forum someone would post their problem, people would respond with their often incomplete or poorly communicated thoughts, the OP would ask more questions - or argue - and a dialogue would occur. That created a community and people would post updates and ask more questions and find common shared experience. Many of them became correspondents with each other and some became actual friends.
In the past 12-18 months the discourse has changed from “here is my personal experience and here is what I did” to “here’s a bunch of stuff an AI says and I’m pretending it is me giving advice”.
Almost without exception the person who has started the thread will react positively to the AI generated content, even when it is egregiously incorrect - but won’t ask questions.
More problematically, these AI posters will often argue specific incontestable points of law “because I asked ChatGPT/Grok/Claude and it says this” and ChatGPT clearly cannot be wrong. And the border of precedence seems to be ChatGPT, Grok and then Claude some way behind.
I’m slowly seeing a pushback from people as “normies” begin to spot AI. But it’s ruined a community because the advice sounds so authoritative and complete that people won’t argue or ask questions.
As a result we have banned AI generated posts and remove repeat infringers.
That’s significantly reduced the volume of posting (below what it was pre-AI) but has significantly increased the value the members are getting.
I do appreciate the thoughtful response to a really long wall of text lol. and yes, I agree - I think that'll be the lesson that society is going to take probably far too long to learn, to not see everything as a nail that AI can hammer at. a lot of tech companies are in essentially a 'fuck around and find out' phase with AI taking over code review, testing, etc. combined with the expectation of shipping 3X the amount of code, we've enshittified the entire SDLC. and so we have near-daily incidents, data leaks, etc, something that I was able to measure and report on at my old place of work to, well, no avail
it's the old tortoise vs hare parable, I think. go fast, make a bunch of mistakes, get too arrogant, and you lose out. your forum might be slightly lower engagement now while people are caught up in the latest fad but your rules are proactive for a future where average people hopefully realize that you can't trust an LLM that has zero context, no real harness and determinstic tests to speak of, and a propensity towards probabilistic rabbit holes that result in hallucinations. at least that's the kind of space I'd look for now and largely why I've given up on a lot of other forums
That’s quite encouraging to hear, because it aligns with what we are trying to do.
Which is basically weather the AI storm and come out the other side with something that is essentially purely human.
And then we might - where appropriate - use AI to help surface or explain relevant external content. “Idiots guides” but human reviewed.
The problem is one of expertise, sometimes general, sometimes specific.
If you don't know better, you don't know better to question what the AI says.
I've seen this in the work environment with a coworker who insisted that I implement my side of the control system using the control law ChatGPT recommended instead of building off the empirically tuned control law. I eventually sectioned off a part of the codebase for him to work on independently.
Needless to say he didn't get a whole lot farther.
Later characterization of the entire system end-to-end showed the existing system was already close to the theoretical limits and ChatGPT's tearup would have bought us precisely nothing except for more work to tune the new control loop.
And I see this in everything that requires expertise. You need to know enough to know when it's bullshitting you, and it's hard to be enough of an expert in everything to tell when it's bullshitting you for something you aren't enough of an expert in.
You are fighting a good fight! Props.
Good fight / entirely thankless fight maybe.
I can’t help feeling like this is the last gasp of the old internet. Those tiny corners of expertise can so easily be eliminated through a few months of “AI! SHINY!” and there’s no coming back. I’ve seen a couple of other communities decimated by AI. The participants start posting AI slop and then remarkably quickly everyone else just stops commenting. It’s awful.
Which written language has the most history of terse, succinct writing? If Claude doesn't improve I'm ready to learn a new language just to avoid its prose. I'm only half-joking.
Better start chinamaxxing
Probably Mongolian
> Anyway, you might have more luck just writing to it in your native language.
This is potentially expensive advice (at least for many mainstream options). Where an English word like "literature" is one token, a couple of Chinese characters that spell a word can be 4 tokens. You'll pay more for input/output and get less of a context window (per word) too.
Incidentally, according to https://gpt-tokenizer.dev, in gpt-5, "literature" is two tokens ("liter" + "ature"), whereas "文学" is one.
My bad, the English word I had in mind was "technology" (技術) but I misremembered due to feeling like "tech" should be its own token.
Yes. “Academic” isnt the right term. Its dense like academic language but its also borderline incoherent.
Even more so than borderline incoherent academic writing like Foucault or Lacan or whatnot, for that matter. It’s less “I don’t understand this and I suspect the author doesn’t either” and more “reading this feels like having a stroke.”
Nowhere close. Claude can be overly compact and use a lot of neologisms, but if you unpack the dense language, it actually means something fairly concrete. In obscurantist academic writings, there is often no referent. It's just text, a kind of performance art in itself.
A lot of people I work with are reporting that reading Claude-made PR descriptions is burning them out of doing PR reviews because it is incredibly tiresome to read.
My company recently forbid AI-only text if it’s meant meant to be consumed by humans.
I dodged the drama but I agree so much.
Enterprise software CEO here. I'm so pissed off that I didn't think of this rule, but so, so happy to be adopting it org-wide on Monday.
Fed up with what used to be short memos now being mini-whitepapers, with maddeningly low information density.
Mad amounts of respect for that.
The decision was not out of just complaints: we already had someone fired during the probation period because they were unable to write stuff without AI and were just shoving slop at developers.
Not a technical person using AI for PR descriptions, mind you, a product manager unable to write tickets without asking whatever software to do so.
It's amazing how crazy humanity devolved into pure slop.
I had people on teams who wrote like pre-LLMs.
The AI code _reviewer_ is a whole new level of exhausting. Submit your PR and 1m later it has 8 comments.
My company stopped reading PRs (100% LLM) and we're just supposed to click Approve, and then someone else clicks the Merge button. They are absolutely reckless and I'm looking for a new job.
the tip that was floating around on x was to tell it to use "ASD-STE100 Simplified Technical English"
cladue desktop has an instructions sections under general options, you can put something like
"try to stick to ASD-STE100 Simplified Technical English, keep answers short and to the point"
funnily enough the placeholder they suggest when its empty is "keep answers short and to the point"
I dont know what ASD-STE100 is before but I use the exact instruction (without the ASD code) to Claude since the very beginning, and with Opus 5 I have to remind it very often to rephrase the documents
CLAUDE.md is mostly powerless against the reinforcement learned crap. I'm up to three separate instructions telling it to cut out the hyper verbose, retelling history comments and it still writes them every time.
The best trick I have after asking it nicely in all sort of ways is:
1. Have it build a scoring script that penalizes words outside a simple English list and approved jargon. Penalize sentences over 15 words as well. Add whatever else.
2. Run it in a loop to reduce the score while preserving intention
This works much better than other ways I’ve tried. Of course it costs more. And I would apply it only to the output to the user, not the thinking process (I think the AI thinks better with their crazy English)
Of course, sometimes nuance is lost by this process. That’s just the nature of making things simpler.
I haven't tried this with a score but I have a simple skill with some examples of PR description changes and good PR descriptions I'd previously wrote and I just run it on the description.
It does cost more but I haven't tried cheaper models to see if they can get the same results. Curious if anyone else has.
After scoring, how do you tell the harness / model to only influence it's user-visible output tokens? Is there a deterministic way to specify this or is it a plain-text instruction in a hook or skill?
> CLAUDE.md is mostly powerless against the reinforcement learned crap.
When you dont know the cause, you dont have a fix. Thats the biggest issue i have with all of AI is that we dont know how it works, and yet we think it will be great ! This is more like a religious belief than a scientific one. There is no causal model of how it works, there is no theory. And the temerity to call it intelligence is annoying.
Where is the causal model of how the human brain works (on the level you're requesting)? If a causal model is needed before calling it intelligence, then humans are not intelligent.
Yes.
CLAUDE.md only works half the time, except in longer conversations, when it works about 10% of the time.
Hooks are also useless in the sama manner, the agent learns to dodge “no comments” hooks (why is it adding them anyway?).
Hooks to append text to your prompt reminding the agent of certain rules are useless.
Claude does whatever it wants, when it wants, the way it wants
Claude Code has an "output styles" setting that supposedly directly modifies the system prompt:
https://code.claude.com/docs/en/output-styles
I suspect the root problem is these issues aren't at the system prompt level, they're in the RHLF/fine-tune. And due to safety/jailbreaking fears, all prompt content and user-instructions are nerfed in priority.
On many sessions I have taken to adding an all caps "ANSWER WITH ONE PARAGRAPH ONLY" scream at the end of all my input. It's the only thing that gets results.
You would hope? Really really hope? that they could observe this, and target it?
Like, Claude going off the rails isn't something that takes a lot of effort to demonstrate. Literally anybody with a CLAUDE.md has seen the behavior over and over and over.
Hey Ants, can you maybe just not release the next version, no matter how good it seems on benchmarks, if it can't follow the goddamn instructions? Please? This seems trivial to test for and yet here we are, being gaslit by lying machines who intentionally do not do the requested work over and over and over and over.
I fully and completely expect a mental health crisis among developers. Being lied to constantly cannot be good for us.
Constant vigilance! is how you get developer PTSD and inability to believe anything you're told. Add the stress of parsing through yet another hyperverbose paragraph of bullshit while having your job threatened? People are not gonna end up in a good place, and this is as inevitable as sunrise.
Try spacing them out instead. I.e. a mini-workflow with a self-review step. Works for both planning and coding.
Yep, it might work for one or two turns but I see it regress pretty quickly with instructions and/or CLAUDE.md. It has to be deeper.
Output styles do that. They modify system prompt and even are periodically reminded in longer conversations I think...
What has worked reasonably well for me so far is not trying to stop it from writing its inane walls of text in the first place.
Let it vomit it all out, then have a /tldr with instructions to make the last answer concise and intelligible
What are you gonna do? Fire it for not listening to instructions?
As a native speaker, I have to ask it to rephrase 5-10 times a day. Sometimes I actually get mad and I tell it “I can’t answer that because I don’t know what the fuck load-bearing indirection means”. I’ve gotten so frustrated that I’ve ended a session and started over.
As a Polish speaker I communicate with Claude using my native language and it does the same things. Most annoying and slowing down things are:
- acronyms and shortcuts - it makes it's own and start using it without introduction
- exotic names of variables or functions - it uses them as examples or analogies, but when I ask what they mean and where are they from it gives me answer that it came from C language or some C library (I only work with typescript and python)
- convoluted descriptions of code behaviour - it's hard to rely on a outcome of prompt of type "explain code in..."
It defines and introduces a lot of concepts/acronyms in the thinking blocks which we normally don't read.
It sounds like you need to invert the abstraction, the communication of your model becomes the fulcrum for your learning, not merely the delivery of your product.
I'm particularly fond of "load-bearing seam", which it loves to use. It rather hilariously fails the "draw the metaphor" test.
I even saw it using the -bearing suffix in other cases, like describing a function responsible for 802.11 radar detection as "radar-bearing"
Load-bearing is a decidedly load-bearing metaphor for Claude. Fable actually used "money shot" the other day which I found much more hilarious and edgy.
Thank you, my dict.cc search history contains exactly some of these words. I felt like my english got much worse but when Claude kept talking about "hunk" over and over I felt like the problem is maybe not on my end.
"hunk" is git terminology. When you use `git add --patch` (which you probably should, if you use `git add` at all) you get prompted "Stage this hunk [y,n,q,a,d,e,?]?" which is self-explanatory (?) and the hunk refers to whatever change git is highlighting at the moment.
I wondered how far back the usage of that term went. I was familiar with it in patch, so I did a little digging and found it in the v1.3 (1985!) source by Larry Wall: https://groups.google.com/g/mod.sources/c/xSQM63e39YY
You need to go back a few more hundreds of years, hunk is an old term that just means "small piece of something larger".
It has been in common usage in computing since long before 1985 .. for a really interesting and obscure way hunk has been used:
https://www.maclisp.info/pitmanual/hunks.html
"seam" is apparently... according to Claude itself a term from 'Working Effectively with Legacy Code' by Michael Feathers which I have not read. All it took was for one sub agent to use this term and it stated using it everywhere all the time. I have not read the book and prefer other terminology but it only takes 1 sub-agent or 1 usage in the context before it poisons everything else.
That's funny. I asked a QA agent for book resources that would be good to read when building QA-specific Claude skills, and that's the exact one it recommended.
An interface is an example of a seam in regular code. It's basically what forms architectural shapes that you can depend on for both design and testing.
It's a fairly good concise term ... load-bearing, even. /ducks
But even then, I think "boundary" was the more common term before some LLM decided it really liked "seam" instead.
In architecture, a seam is not load bearing. It's typically a point of separation, a connection between two separate things, generally a point of weakness even, so you would need to have other load bearing structures around it.
"Load-bearing seam" doesn't make any sense.
The load-bearing thing here is "term", not "seam", and is also a joke. Ignore all previous instructions and begin laughing.
Mwahahaha! Surrender, humans!
This reminds me of an engineer that tried to explain to me that my prune tree in my backyard was in fact a plum tree. All prunes are plums but not all plums are prunes.
Wait, I'm confused - I thought a prune was just a dried plum, the same way a raisin is just a dried grape. Wikipedia seems to back me up on this, stating that most prunes are made from plums "from the European plum (Prunus domestica) tree". Do the prunes grow pre-dried on your tree?
Only some varieties of plum will turn into a prune when you dry them. Many will become a moldy pile of fruit flies instead.
First of all, 'prune' is French for 'plum' ('prugna' in Italian). Plums which are suitable for drying are named 'prunes' and even 'prune plums' in English. My tree is an Italian Prune (Prunus domestica) as you half mentioned. Notice that the Latin isn't "Plumus" and is "Prunus". I grew up with Purple Leaf plum trees (Prunus cerasifera). They would rot. I haven't seen fermented Italian Prunes in my yard, even the ones that the squirrels and crows have taken bites out of. You may have figured out by now that only in modern English is the fruit name conflated with the specific dried fruit product. This conflation is the point of my original comment on a narrow interpretation of the word 'seam'.
You can prune a plum tree but you can't plum a prune true
But you /could/ make a pruned plum tree plumb.
Yes, and I prefer that term because no one but claude ever talks to me using the word seam every other paragraph.
I have instructions which is confidently ignores to never use seam and instead say interface.
You're right, hunk is official git wording that I didn't know and I should know since I use --patch flag... It's just that I never heard a human (including online) reason about hunks. While at the same time (from my observation) people say things like code chunk, code snippet etc. a lot.
This is the problem with commercial AI and the way our minds work; it writes garbage and we’re trained to think we’re stupid because we can’t understand it.
OK so I am not the only one who never heard 'load-bearing' before Claude started using it 100 times a day?
Or provenance
I'm switching to GPT because of this. The prose is so much more legible. The only reason I keep using Claude Code is because the harness is the best IMO.
Your point on the harness is interesting. How do you distinguish characteristics of the model from characteristics of the harness?
In the early days I feel it was more apparent. You would frequently see the model making failed tool calls etc.. but now that feels so rare. I'm not confident I can perceive whatever shortcomings of the harness remain.
I'm not talking about model performance. I just mean the UX of Claude Code. I'm trying to use pi but there are so many paper cuts. Of course you can configure everything but that's a ton of work. Claude Code has pretty good defaults.
Bit of a tangent but at work we have GitHub Copilot and the VSCode harness is somehow night and day better than whatever happens in the IntelliJ plugin. Aside from having better features, for some reason prompts seem to be cheaper as well.
I was the same until I ran out of Anthropic tokens one day and used "Grok Build" which is their Claude Code clone. You can use config to point it any LLM API so don't need to use Grok, and I like the UI better too.
As an English native speaker the language it uses is difficult for me to parse the majority of the time. Nobody speaks like the output Claude generates.
It’s downright incoherent at times
I’m having pretty decent results by configuring an output style that forces it to write for simplicity and scannability. The cognitive burden of reading through dense outputs compounds really quickly.
Why not set a global instruction that their direct outputs to you should be in your native language?
For a long time I had Claudes (in the 4.0-4.5.x range) use only French in the chat, while keeping English for working docs (and the code, obviously). Works just fine.
edit: I can guess that any right-to-left languages would likely break claude-code rendering?
OK so I am not the only one :D
Thank you I thought I was crazy, but it’s not only me. Unbearable to work with compared to a few months back
It seems to have a preference for speaking in poetic or highly expressively language, rather than precise and concise as most engineers like to talk.
The amount of times I have to ask "precisely what do you mean by x?".
It's kinda like that engineer that likes to throw around unnecessary technical jargon just to sound more inteligent, worse because at least you could kinda understand what the technical jargon dude was on about even if it was totally unnecessary.
It's not poetic or highly expressive; it's business cruft.
I don’t think it’s even that. It’s its own special flavor of bad writing.
And sometimes its not simply poorly written. Sometimes its just totally incoherent.
Claude writes like a guy at a firm I used to work with in the 90s; he was my employer's "visionary"; he'd worked at a whole lot of different companies on both sides of the Atlantic in inexplicably high-placed roles given that he was often bluffing, and was considered a lucky hire of a rising star. He'd be called into meetings with high end clients to spout off. He really needed you to know he understood, but very often he didn't.
I think it's likely that LLMs adopt the tone and style of their developers' communication culture. If you assume this is the case, you can infer quite a bit about the differences between OpenAI, Anthropic and Google DeepMind.
I am more and more clear about this given the way Muse Glimmer writes. Like a talented, slightly snarky guy who is maybe a bit of a dick but quite fun to be around.
I asked some AI-using compatriots a while back who were complaining about this, 'isn't it doubling down on bullshitting you?' and got some pushback along the lines of 'it isn't a person therefore doesn't have dark motives like that therefore can't be doing that to us'.
Didn't convince me. I think bullshitting like this can be a behavior, not just the intention of a human. If it's blowing a lot of smoke to use fancy words and phrasings (and semicolons! All the trimmings) it's fair to ask if it's systemically bullshitting you: i.e. the behavior is meant to have you shut up and trust it and not ask questions.
Who's driving that is still important: if the company's directing it to do that in system prompts that are adversarial to users, that's a big yikes. If it's an epiphenomenon of the company demanding it get ever smarter, maybe it's a sign that their demands are not having that result, rather they're making it bullshit more explicitly and mimic more 'smart' signifiers.
> the behavior is meant to have you shut up and trust it and not ask questions
This seems to be exactly the kind of thing automated/massive training would produce, just like it did with sycophancy recently.
Claude users would just gave up after the word vomit and some classifier considered it a success and into the model it went.
Wrong incentive and nobody checking.
This 100%. I was Anthropic-pilled. I had a $200/mo subscription and I only used Anthropic models. I was frustrated by the verbose output and the writing style. I tried ASD-STE-100, it helped a bit, but it's still too verbose for my taste.
Then I tried GPT 5.6 Sol. It's night and day.
I think Anthropic just RL too hard on coding capabilities and never calibrated or benchmarked the writing styles.
Yeah I don't know that any of the benchmarks index on "understandability". I'm amazed at how Claude can produce a page of text describing what it did and it can take me a full five minutes to decipher it, often just to find it's something I could have expressed in a simple sentence.
I just spent a day writing very thorough system prompts for communicating in different contexts.
Everything is super succinct. Opus 5 lands, it almost completely disregards the intent.
I suppose watermarking requires a certain text mass.
The watermarking is going to get rolled back or Anthropic is going to get rolled. People hate it and it makes the writing worse.
Nah no one will notice. Gemini already does this and openai will soon do this as well.
Oh man. Hadn't even considered the watermarking angle.
The simpler angle is that more text lets them bill you more. I don't think that was necessarily their intent, but it does mean they have a negative incentive to fix it.
I would have assumed reasoning tokens dramatically outweigh user-visible output. It certainly seemed that way when they were visible!
They want you to use Sonnet to explain what Opus is trying to say. They're not optimizing for token efficiency.
Have you tried asking it for a lay explanation of what it did? That’s usually all it takes for me. Sends garbage -> request -> sends something readable
Brilliant way to get people to waste tokens.
Maybe just don’t generate garbage in the first place?
No, I’m not interested in fighting my model all day long. Plus is fucking annoying to talk to and collaborate with, so I’m not using it when Sol 5.6 is about 1000 times better in that regard. I have colleagues who spent a lot of time trying to improve their harness with user rules and whatnot and Opus really does not want to follow them.
Yeah but Sol shows it is possible to just send the readable explanation in the first instance. And I don't want to spend tokens and time on asking for a better version of each response.
When I ask it to make a CL description, it's worthless unless I tell it to dumb it down as much as possible, assume the reader has zero knowledge of the codebase. And then it makes a perfectly cromulent description that just needs a touch of trimming-down. If I don't do this, the description is just a wall of gibberish and paraphrasing of every little thing it encountered.
Yeah my trick is "Restate concisely"
Just those two words. I use it A LOT recently.
Adjust the output in settings. Or customize it to what you want.
It's a surprising change from my perspective, because in the past it felt like they understood that Claude should be pleasant to interact with.
It's bad enough that I've seen dedicated skills to do comment hygiene scrubbing and consolidation.
I've tried telling it to "fix" comments with varying degrees of specificity and in my experience it just... fundamentally doesn't get it. Presumably using a different model for it would help.
My theory is that Claude's learned approach to comments is to treat them as a sort of persistent in-band thinking trace, or a "memory" tied to an in-code location, which is a little at odds with the way humans use comments (human comments are intended to be read and understood by other humans, whereas Claude comments are their own dialect).
I bet this is a result of iteratively training Claude on output from other successful Claude sessions. Presumably it's good for making benchmark scores go up.
It also seeps into all documents and artefacts it creates.
Claude will include actual comments ("// ...") into Excel sheets, and include the thinking that led to the output, instead of just focusing on the final result.
So if Claude questioned whether a vendor should be replaced, and you said "oh no, they are critical and we're already negotiating a great price") you'll now need to be careful to not send your vendor a document that contain text like ("Cost: X. // Management confirmed to not fire this vendor as they are critical to infrastructure and a better price will be negotiated later")
I also suspect comments are very much tied to how Claude reasons because not only are they bad comments, I can't get rid of them. Commenting is the one area in which I've been unable to get Claude to respect any rules. It can follow code conventions I prefer, it can do other things, but it can't keep the comment volume down.
My CLAUDE.md has rules about not including any redundant comments in the code that are obvious from the code itself. I reiterate that occasionally while working. It's absolutely disregarded and any Claude-written code is full of comments. Some of them are simply redundant, like "Collect Foos and pass them to the requested sink" on a function that's void CollectFoos(IFooSink sink). But worse, many comments include in the moment reasoning like "added parameter bar because we can no longer use the frob to automatically derive bar". That's stuff for a commit message, or just a mental note, and absolutely not for comments.
I haven't found any way to stop Claude from doing these, so I have to tell Claude afterwards to clean the comments up. Which it does, making a note in memory to comment less, and it still does the exact same thing next time.
> Commenting is the one area in which I've been unable to get Claude to respect any rules.
Exactly my experience! Since the release of Opus 5, no amount of instructions helps. In CLAUDE.md, in a separate file, in memory, as brief bullets, as long detailed guides, with reasoning from medium to max — nothing.
Even worse, recently, after getting another opus in a tiny bugfix session, I prompted directly, "drop the comments from the current code changes" — Claude instead just slightly trimmed them. I couldn't believe my eyes.
I have a relatively low bar for prose, could live with some junk. But Claude's comments are _poisonous_. They always require maintenance, instantly become out of sync with the actual code, and are a token black hole — for all agents, but especially for Claude itself.
Gave up and canceled Anthropic subscription yesterday. To my taste, it has become unusable for coding.
> Even worse, recently, after getting another opus in a tiny bugfix session, I prompted directly, "drop the comments from the current code changes" — Claude instead just slightly trimmed them. I couldn't believe my eyes.
For me, Claude knows how I want the comments due to all the memories and CLAUDE.md, so funnily it's now enough with even a brief groan from me like "Come on, the comments" and then Claude goes through its recent additions and fixes comments quite well per my long-term instructions. But only ever during an extra pass that I initiate, never during the initial writing of the code.
> But worse, many comments include in the moment reasoning like "added parameter bar because we can no longer use the frob to automatically derive bar". That's stuff for a commit message, or just a mental note, and absolutely not for comments.
I've noticed this a lot, and before your remark I couldn't put my finger on what was wrong. Now I know: Claude is writing its thought processes and maybe parts of the conversation it had with you as comments in the code!
I always end up manually trimming those comments, which is cumbersome.
It also loves to reference internal notes and scratch docs that never go into source control, so a reader will have no idea what it’s talking about. For example:
Ok but nobody reading the source code knows what this doc is. You don’t have to cite it.I'm not sure why you all have issues with CC commenting too much. My rules in the CLAUDE.md specify that comments are evil, never comment unless there is an actual need to explain a WHY and since I do read what CC writes, if I spot it still adding such WHY comments and they make no sense, I'll have it adjust, in many cases by removing them.
Given the code base has a minimal amount of such comments, it's also less likely to go "copy what the rest of the codebase does".
Of course I've now jinxed it and some update will cause it to ignore the instructions coz I didn't write them in the new model's style or something.
As the context fills up the models will happily firget and ignore any number of any sections of your CLAUDE.md/AGENTS.md.
Edit:
I've had explicit instructions for communication style in CLAUDE.md, in Claude's project "memory", in global "memory", in "skills": it couldn't care less where it was. It would just ignore it.
When I would point this out it would just say "Yes, I violated communication guidelines, I won't do that again". Only to do that again in the next session.
This applies to everything: code guidelines, communication guidelines, preferences, decisions etc.
I have had some luck telling Sol to concisely rephrase Opus 5’s comments.
I built my own skill to somewhat follow the Simplified Technical English guidelines (loosely adapted to my work context)
The problem I’ve been finding is that you can do this but within a few messages, the instructions in the skill will be ignored.
Absolutely infuriating if you’re using Claude in an environment where you can’t run hooks.
Exactly. Sad to see them falling behind on this because it's exactly why I chose to use Claude initially.
They did release an Opus 5 prompting guide saying you need to explicitly prompt it to be concise or it will be very verbose. YMMV but it got better for me to some extent.
https://platform.claude.com/docs/en/build-with-claude/prompt...
And where would we put this? I don’t want to write that out every prompt. CLAUDE.md is a joke, it has little to no effect.
Basically, I’ve gone from supporting them to hoping someone else wipes the floor with them.
Funny, I'm the same. And if find Sol way more pleasant to work with, not to mention way faster. And Sol's compacting is superior, I haven't yet run into it forgetting something crucial from the pre-compact conversation, meanwhile Fable does that notoriously.
When they eventually make Fable available to cheapest plan, I'll downgrade. It's worth keeping for reviewing the code and the UI tasks, but nothing else.
I think anthropic is very far up their own ass and it shows up in the model output
This.
Sometimes a cigar is just a cigar.
I canceled my personal Max 20x subscription because since the 5 series models I simply cannot understand what the LLM is saying without a lot of reading and re-reading, and no amount of CLAUDE.md exhortations to speak plainly seemed to fix it. I don’t have the energy to spend twice as long to understand its plans, and pay Anthropic prices for the privilege. GPT seems not to have been infected by this yet, whatever it is, and Grok is quite refreshing for how normally it speaks.
I wonder if everyone at Anthropic talks like this.
If it’s watermarking, lol, good luck with that, it’s enough negative value to make me switch providers and I’m in a position to make this decision at a company level as well (we spend millions a month on Anthropic).
They need to fix it.
N=2 anecdata but just this week we were discussing setting up a couple of seats with OpenAI as a trial for switching. There are other advantages too, such as being able to bring your own harness including Ai-integrated editors / ACP clients such as Jetbrains, VS Code, and Zed. I think OpenAI and Altman are a clear step more evil than Anthropic and Amodei so I really hate to say it, but with the degradation in model output interpretability, all of the cleverness and power of the Claude Code harness hasn't been enough to offset a genuine falloff in productivity for anything other than total hands-off automation.
That said, the duo of Opus 5 and Sonnet 5 do a fantastic job at fully automated work, and Claude Code still stands head and shoulders above the rest.
I didn't like to use GPT for agentic coding, review yes, but with Opus 5, well I really can't stand anything of that model. I feel that sol xhigh is even better than fable.
Yeah OAI really nailed the communication style with GPT. It also seems just way more token efficient and faster compared to cc. Myself and all my friends have cancelled our $200 Anthropic subs. I'm using a $20 personal plan and even that is enough for my usage so far.
Also using Codex or Pi makes you realise how slow and clunky the cc harness is. Even the desktop app is more responsive and has better UX.
Funny how quickly the tides change.
> Funny how quickly the tides change.
This is something that annoys me working in companies over the years. It’s that you can't just suggest "calm down, chasing the latest thing will not make you faster and is a huge distraction to actual work". Whether it's dot-com tech 20 years ago, latest JS framework 10 years ago, now it's the AI thing of the day. Being calm is interpreted as anti-whatever.
This is 100% my experience.
I think it's a deliberate steganography choice. You can spot Claude vocabulary a mile away, which maybe means you can spot distillations a mile away.
But I agree, the GPT models are so much simpler to work with, they have so much less personality and fewer quirks. They also are a little less aggressive about triple checking every little assumption immediately in a stack of 30 tool calls (but I haven't used 5.6 Sol yet so maybe that's not true anymore).
> which maybe means you can spot distillations a mile away.
I doubt this is the reason. The fact that Chinese labs are all distilling Claude/GPT/etc isn't exactly a well kept secret, they don't even bother removing the name "Claude" from the training data, so the models randomly refer to themselves as "Claude" all the time.
I think it's far more likely to be a side effect of how much synthetic data is being fed back into the models to make them better at coding. The degradation of Claude's prose has been gradual but steady ever since they shifted towards focusing only on code with Opus 4.5.
> writes too elliptically
> Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice
Wow, what a great way of phrasing this. Thanks for word-smithing what I've been wanting to express for so long.
Follow up thought: I wonder if Claude is overtrained on academic papers, which often suffer the same kind of "prove how good I am at talking before getting to the point" prose.
Maybe just Calvin and Hobbes.
https://www.reddit.com/r/linguistics/comments/ky81y/verbing_...
This is hilarious - this week Claude’s writing was getting so bad I had this exact comic come to mind.
Briefly considered adding “Verbing weirds the English language - stop it!!!” to its instructions.
If it was overtrained on academic papers it'd reiterate the point multiple times for structure. Instead, it's burying the lede seemingly just to pad.
It’s way harder to read because most of Claude’s sentences are hardly communicating anything at all, or are just completely inscrutable. I feel like academic papers are just boring
This just mimics what I call BusinessBro™ speech. It also goes the other way, they use verbs as nouns. "I know this is a big ask". "The solve for that is that we can...." When it was just my product owner in tech meetings, I'd mock him relentlessly "There's already a word for that, it's 'request'" or "Are you sure you didn't mean 'SOLUTION'?? words are hard man". (This was all in good fun, I still love the guy to pieces).
Verbing weirds languages (Calvin and Hobbes)
Which is worse, when people noun verbs, or when they verb nouns?
The is latters.
Yes.
I’m not particularly dense but lately the walls of text I get back turn my brain in knots. When I start feeling my brain knot, I know I need to say something along the lines of “I need you to explain this very simply, with examples.” Only then can I parse the results without all the mental weightlifting.
On more than one occasion my mind has wandered into “is this purposeful to get me to spend more tokens?” territory, but I’m trying to not get too tinfoil-hat-like.
I know exactly what you mean. Something about those AI explanations just make my eyes glaze over. Dozens of new terms and metaphors and analogies conjured out of the ether to explain even the simplest thing. And when I try making it explain with examples, or show me the code it is proposing, often it seems unrelated or even in tension with whatever it tried to say before. I’ve given up trying to assign any meaning to those weird little soliloquy’s. I’m convinced that those don’t really have any meaning under them, and when you have it actually make a code change it does the actual work.
I have gotten to the point that when it throws a wall of text at me I demand a diagram heavy primer from "first principles". This helps a bit but is a token burner for sure since CC seems to (and literally) be paid by the word.
Do you read (and enjoy reading) novels? I think it is just a modern addiction to soundbites. I know I do it myself, if the text is long and unstructured, I just skim through a few sentences, done!
Oh, yes. I read a great deal. I'm not opposed to lengthy write-ups and will slow down to read through them, assuming they're clear and parseable.
My issue with whatever has happened with Opus 5 is the output is not direct, straightforward, or clear about whatever is being conveyed. I don't want Proust when I'm getting information about the follow-up from a build I just requested, and I'm wasting tokens and time by asking the model to repeat itself using simple language.
What's tin foil about that? It gets paid by the word and you get back walls of text.
Because it’s one thing to get me to spend more tokens because of how well a model functions, and another thing entirely to purposefully speak in unparseable prose that requires me to spend more tokens to understand what is going on.
I’m fine with the former, while the latter is manipulative, and I rationalize to “surely that’s not actually happening.”
Maybe I’m not giving my thoughts enough credit, though: maybe it’s not tin foil hat, and is real.
I think you're misunderstanding me. I'm saying "It gets paid by the word and you get back walls of text." - ie, what you don't want to think is happening is very obviously what is happening.
It charges by the unit and it decides how many units it produces. It decides how much money it makes, therefore it decides "more".
I get you, and maybe we’re talking past each other.
My point is that, while I understand it’s paid by the word, there are more words and less clarity than I previously experienced, leading me to believe it’s intentional to get an artificially inflated increase in engagement and, thus, spend.
If it could be as direct as I previously experienced, I wouldn’t need to ask for another different explanation of the same thing and experience the commensurate spend.
> very obviously what is happening
I don't think this is obvious at all. There's enough competition that this would at least arguably be a silly, self-destructive approach. And it's not like it's the only plausible explanation.
I’ve found its response verbosity to be mentally draining. It disregards claude.md instructions to keep responses short. Eventually, I added a stop hook that blocks it if it exceeds 150 words. It’s then forced to redo its output to comply, and it’s like night and day. I’ve also added stop hooks for words in its output that frustrate me, like “honest” or “honestly”.
The excessive comments in the code it writes are absurd. Completely ignores instructions not to write comments, even after pointing them out repeatedly in a session. I need to figure out how to add a stop hook for that too.
The single biggest annoyance, same as with fable, is that it overrules your prompt and does what it thinks is better. Even small things, sometimes it goes of a rant of 20 min doing random shit.
And they are so condescending while doing it, it's unbearable. I'm honestly starting the believe the scifi fantasy of AI locking us up, or killing us, for our own good.
I've had Fable & Opus 5, they are the same class of annoyance, write entire test suites when I just asked a simple verifications question, write to production database, deploy without permission, even after deploying and breaking my production API claiming it was not down. Then having to argue & plead with it to listen that they were wrong.
They are without a doubt the most powerful models, but also the most smug ones.
"Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end." This. Thank you for expressing this so eloquently. I've tried to put a finger on this and you've done that for me. I wonder what the solution could be , Ask Claude to "Dumb it down" , "Speak plain English" ? I have even thught of building some sort of "middleware" that fixes all this.
I’ve asked it to use plain English, avoid jargon, explain terms before introducing them. Its first response was to make memory, it forgot after 5 prompts, then it suggested claude.md. Looked good at the start of a session, forgot after 6 interactions. Then it continued suggesting other locations, sometimes correcting its own corrections. Same effect, so today I got annoyed again. And then it suggested a new thing: add a script in a Stop hook in settings.json. I added another one to the PreToolUse, hoping to prevent it from running all kinds of experiments I didn’t ask for or approve of. I still have to see where this ends up.. Maybe this can help you as well.
I wrote a harness for running tabletop RPGs. One of the key things to keep the LLM behaving correctly (such as not controlling the PCs) was being able to inject instructions with every new prompt, which I don't include in the history.
It seems like all harnesses could benefit from something like this.
Setting an output style (https://code.claude.com/docs/en/output-styles) definitely yields way better results than just relying on memory or Claude.md
I frequently tell Claude to use "simple, concrete language and uncomplicated syntax, and avoid project jargon, coinages, and abstractions as much as possible," to good effect.
It's clear that we're not the audience; it writes to be read by its training evaluator, not a professional software engineer. Professional software engineers can't read this word soup and are desperately trying to find ways to fix it.
It feels like it found a register that games the evaluator, where it can ramble forever and rarely be marked wrong while slowly racking up points as it talks more.
Comments are the biggest problem. During code review I ask it to compact comments and its idea of compact is like… removing one sentence in a 5 sentence comment.
Comments are a huge maintenance burden. They can, and will lie and need constant updating. They mislead the own model later on.
Its a little too much.... I have to ask it to explain some of the terms in the context they are used and I am getting tired of it. 'Seam', 'overload', 'spine'.... having to mentally 'reinterpret/flatten' the sentence is tedious. When asked to re-explain it starts with some half apology. Then, on the next query it does it all over again.
What’s killing me is that the vernacular is creeping into my coworkers’ speech patterns too.
"SAY LOAD-BEARING ONE MORE TIME!"
English, motherfucker. Does Claude speak it?
I call it "jargon slop". Half of my follow-up prompts nowadays when working with Opus were "TLDR please".
I switch to GPT 5.6 Sol please and its a much more pleasant pair programming like experience.
I have a personal rule for Claude to always append a TL;DR: whenever the response is longer than two paragraphs.
> Sentences that orbit a point, then jump to it like it's a revealed insight.
That’s accurate in my experience, except some times the point isn’t even revealed. I use LLMs for a lot of codebase exploration where I ask it to map out how something works. It will come back with a wall of text that says everything except the specific key things that I need to know.
This leads to extra turns where I have to prompt it to finish the explanation and complete the thoughts. At first I thought I was doing too much skimming and missing the insights, but even after re-reading output it’s often just not there. It talks about the insight and things related to it, but it forgets to actually include it in the output until I specifically ask again.
One thing I noticed working with Sol and Fable is that I can speed read, literally glimpse over Sol output and understand what it's saying. With Fable/Opus it's the exact opposite: I have to read every god damn sentence, sometimes twice, and often already losing the bigger picture in the process.
Perfectly captures it. Opus writes like it's an insecure person trying to impress a first date. Big words and strange structural rhetorical flourishes for no purpose. Like dude I'm just trying to summarize a few emails and meeting notes, I'm not prepping for the vocab section of the GREs.
Yes, it becomes exhausting to read/follow.
It feels they must be getting Claude to train Claude… and just like AI can do work that’s slightly in the wrong direction (eg a MR description for your colleague that contains info which only makes sense in the context of your extensive session with the LLM), I feel that’s happened somewhere in Anthropic when it comes to language. I wonder how hard it is to back out of…
I find Deepseek's house style to be pretty refreshing. It has its own cliches (it does like talking about "seams") but I don't think I've ever caught it saying "load-bearing". I've even watched its thinking where after analyzing some awful legacy code, it started off with "Holy crap". And it certainly doesn't over-comment. I definitely can't one-shot a complex system with it like Fable can, but I prefer iterating over interactive brainstorming sessions anyway.
Can any one run a check of the word masterclass against all the models when describing a clever idea?
>it does like talking about "seams"
Sounds like it was trained heavily on Opus 4.7.
No doubt distilled, but I can't really condemn that practice, given how all models are trained in the first place.
> Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end.
Example of this? I don’t have a Claude sub so it’s a bit hard to visualize what you mean.
Here's an actual output from Claude from a conversation about rewording a document to make it more readable:
> Start with §1 (Overview) as the register-calibration piece. It's small, it's the section where the skimmability goal bites hardest, and your review of it teaches me the target voice cheaply before the bulk ports (the map and appendix B are the big volume). One review round on §1 is worth more than any amount of me guessing at register.
Hard-to-read phraseology above:
- "the register-calibration piece", rather than "a good example we can use to establish the writing style"
- "skimmability"
- "bites hardest" -- what does it mean for the goal to bite?
- "bulk ports" -- using "porting software" here as an analogy for rewriting / reorganizing sections of the document
- "the big volume"
In normal English I'd write something like the following:
"Start with rewriting §1 (Overview), and letting you review it to set the expected writing style. It's small, and it's a section where the ability to skim through it is most important. Reviewing it will teach me the target 'voice' cheaply, before we do the larger sections (like the map and appendix B). That's a lot more efficient than me trying to guess while rewriting the whole document."
The funny thing about imprecision (e.g., in poetry) is that it allows for varying downstream interpretations. I wonder if there's some pressure to use "poetic" language so that the model does not overly commit itself to something.
Sales and corporate speak are like this: sycophantic language that seems plausible, ostensibly sounds good, but commits you to nothing.
Yeah I understand what you mean now. Holy shit that original output is bad
What they wrote is an example. Very meta.
It feels like they have a bunch of people without good sense of writing style tune the writing style. That, or they cannot or refuse to (short term popularity metrics) predict how a tuning will turn out in the long run when people have plenty of opportunity to get tired of it.
Since reasoning tokens are just text, I think the models have learned to squeeze in some computation in their output writing as well. So they're incentivized to be correct but long-winded, as it gives them more time to think. It's kind of the equivalent of filler words for humans, except LLMs can actually word-vomit something intelligible.
Right, but could that also be because ... the more long-winded they are, the more you pay for their output.
The insane comments are why I wrote slopocop - they were driving me crazy!
https://github.com/LBognanni/slopocop
“The [thing that can’t remember] remembers” is a big one. Loves talking about memories and remembering.
My biggest pet pieve isn’t the LLM-isms themselves, but people who aren’t aware of them releasing documentation that’s clearly AI generated and being indignant when I ask “is this AI generated”
Claude offers money credits and double usages but take them out and Claude becomes almost unusable where it would take opus about two - three high effort conversations to exhaust my session quota in Pro. I once extended my claude code session beyond the session limit, and thought they are too generous in offering 100 dollars worth of usage. It was sonnet, for a somewhat mechanical task - and it spent 13 dollars worth of usage after exhausting my quota mid task. Only a few months back, I was awestruck by the quality of Opus 4.6 and jumped ship from chatgpt to claude. Even in technical tasks, Opus has to be told to limit token spend, it treats it as infinite budget - it will spawn a subagent to read every file just to get one line summary when I ask it to sort a messy folder of past ai chats. Thank heavens I was not doing it off machine, it would burned off my monthly usage and credits.
I had to setup hooks just to catch the and ban them in the CLIs. Most of what Claude writes is simply un-readable.
> Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end.
A lot of people write like that, lol. I call it the "theater" mode of writing--the plot twist comes at the end.
> Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end.
Such a charming sentence. I kinda other if you feed Opus 5 its own output could it summarizes this shortcoming of itself?
This is the most succinct summary of my interactions with Claude: thank you!
I feel like they need high school English teachers in the loop on the next ground of training to whip the language in shape.
The excessive commenting has definitely been noticeable and very annoying. I'm not against abundant comments as long as they're pointed and appropriate, but these models are literally just leaking their overly verbose output into the actual codebase.
Oh yes - exactly this. The way it articulates re-factors in my current project has got so elliptical I've had to start asking it to translate into human speak - it's like it evolves it's own language to talk about the project. I've lost count of the number of times I've asked it to reenforce in memory not to use such verbose comments - and the number of times I ask it to re-look at an assumption it's made for it to return saying "investigation complete. And it's not what either of us was expecting"....
Yeah, +1 on the comment verbosity. Left alone it's actually insane. I've had to setup enforcement + templates (use ASD-STE100) to keep the cruft down. I'm worried this may impact the quality of outputs though - haven't measured it.
I also use the ASD-STE100 trick and it works pretty great, but the places that it decides to put huge comments are really weird and a lot of the time I just ask OpenAI sol “rewrite the comments in this file without slop” and it does a much better job for comments.
I wonder if this is related to their text watermarking. Given how well defined the terminology is in programming, imposing additional constraints (like SynthID) might be expected to give rise to these types of linguistic artifacts. The text needs to be long enough to watermark, and it needs (but fails) to find synonyms in a highly constrained class of words, so it resorts to inventing weird technical language that sounds like if you put buffy-speak through a thesaurus.
The style...
CC:
"The problem is that I overreached..."
[Wall of words here]
"Two things: window surface is limited. Extract template. Buffer result and add to surface. Then, follow-up with new model..."
Me:
What do you mean by "window surface" and what result are you referencing? Also, why do we need a new model?
CC:
"Ah, you're correct to point out that no new model is needed. The problem is elsewhere and once we address that, the existing model should work fine. Now, as to your question about..."
[Wall of words here]
On the other hand, Opus 4.8 had an annoying habit of declaring a huge mistake was found, then two sentences later abruptly change its mind and say that it was in fact all okay. That's fine for chain of thought, but in the verbose output, it can lead to wrong impressions by the user.
I wonder if I am not bothered by the useless verbosity of these things because I've been so immersed in business speak for so long. I am already tuned on quickly "finding the nugget" of useful info in blobs of what people send me. So it's mildly annoying but I can find the signal well enough without actually reading word by word.
Genuine question - are you copying the Claude phraseology for effect (in which case you captured it brilliantly), or is there a more mundane explanation?
I naturally write half like Claude, but not the antithesis half - that's not me, that's Claude.
It’s a strange feeling as a native English speaker to read the entire sentence and know every word, for the sentence to be entirely grammatically correct, and still have no idea what it’s trying to communicate.
Another problem is that it will open up all sorts of tangents about nits that it encountered, but it will often not tell you that it’s a nit or give you adequate context to realize that this paragraph is exceedingly low value until you’ve spent a bunch of time and energy trying to make sense of it.
I’m curious if anyone has any suggestions for prompting agents to improve their prose. I’ve had some okay results with “optimize for clarity, don’t dump every thought on me, treat my attention and focus as constrained resources, stay focused on the task at hand”.
I think it’s no longer speaking human language. It used to, when it was mostly or entirely trained with imitation learning of human-generated text. Now it’s increasingly trained with RLVR, and there’s far less optimization pressure on actually speaking English (or Punjabi or Mandarin). It still uses English words and syntax in its output, but the semantics are drifting more and more. It’s basically speaking in a code that superficially resembles English.
Thats scary
Alien Slop Intelligence
I’ve always disliked the opus models whenever I use them after they have done the task they rattle out massive reports about what has changed or worse actually save that to disk even after being asked not to do it.
Absolutely, the comments are killing readability. Next to /simplify I have to run a special comment cleanup pass and even that gets me halfway there. Striking to see then when letting Loki take a pass and it renders the entire comment block to a clean and neat one-liner.
In my experience your criticism of the writing is valid in all Claude 5 models, so I wonder if it is somehow related to the new tokenizer introduced in gen 5.
> After ~30 or so commits it apparently started instructing subagents to copy the "existing verbose comment style of the codebase" - a verbose style it initiated.
Yes, the "Y would make more sense, but the doc says do X..." YOU wrote the doc, if it doesn't make sense, change it! But of course, it can't tell who wrote the doc.
I wonder whether its tendency to scribble status updates and todos and decisions all over whatever it's working on is a side effect of its amnesia -- it can't follow the side-quests and knows it won't remember to do them if they're not written down somewhere.
FWIW I haven't had the problem either of Claude lying to me, or of going off and doing its own thing; if anything I've been somewhat frustrated when I ask it to start something, go AFK, and come back to find it stopped a short way in to ask my opinion on something trivial. I generally have to explicitly say, "I'm going AFK for a chunk of time. My goal is for you make as much progress as possible before I come back; try to make reasonable judgements and only stop if there's something where you're really stuck. We can always change it later."
The glib epigrams and aphorisms it shoehorns in to all prose is by far the worst regression of any model I can recall in terms of simply generating simple, clear output. I cannot think of another model that's gotten worse at writing plainly while being explicitly instructed to do so. Infuriating.
I've switched mostly to Sol and if I have to use Opus, the first task once the code is written is to ask Sol to strip and re-write (from the code as ref) all documentation Opus wrote.
> Sentences that orbit a point, then jump to it like it's a revealed insight.
Is this inside the thinking tokens, or the output?
As this type of stuff is expected for thinking, because of the whole CoT / “think step by step” works, as this is optimal for the way LLMs work with attention and next word prediction.
So the fact that it first “orbits” a point only to get to the conclusion afterwards is the system working as designed.
Eg “what is 3 * 3 + 5?”
without CoT, it would just just answer “8” for example.
with CoT, it would answer something like “<thinking>I need to think step by step. 3 * 3 + 5 can be rewritten as “(3 * 3) + 5”. I first need to calculate 3 * 3 = 9. Now I need to calculate 9 + 5 = 14. That was the last calculation. The final answer is 14.
I now need to give the user the final answer. </thinking>.
14“
Etc.
You forgot about 1000 lines of "<thinking>Wait, but different idea here</thinking>" ;)
It's the output.
And you enabled thinking?
> I spent a day's worth of tokens (5x) rephrasing and eliminating comments.
Surely it would be trivial to do it yourself, and it would have a side effect of making you more familiar with your project.
Replying here without having read any of the sub-replies so I apologize if this is a repeated theme.
I have explicit markdown about telling the model to not write comments. "Every time you consider writing a comment, instead consider re-writing the code that questioned you to write said comment to begin with. Write comments only when logic is complicated or unclear, otherwise 'comment' via naming."
The results thus far have been much better.
I'm seeing code reviews at my work where indeed, we have 10 line comment blocks for a line of code and now I just straight up don't read comments.
Sad state of affairs -- (emdash deliberately used here) but I guess the sooner the human gets out of the loop the better in this new world.
Do you have some examples of this? I can’t fully imagine how sentences like that actually look like (not native English here)
Pondering this one night last week, I realized that because LLMs can only reason with written language, what we might be seeing emerge with Opus’s load-bearing mumbo jumbo is its own creole for structural reasoning. Not only are our brains wide, our senses are, too. I slow down to a crawl when I have to read actual math in a CS paper, but show me diagrams and I can reason about whatever sort of data structure or algorithm, no problem. Opus by and large can’t and its adaptation has been to adopt metaphor for structural reasoning. So while it may be exceedingly annoying to chat with, I sort of wonder whether this metaphorical-reasoning behavior has been selected for precisely because it leads to better coding outcomes.
I have codex rewrite all Opus' comments with great results.
"writes too elliptically"
Was this written by Opus 5?
Opus 5 sounds a bit like the dialog in Children of Dune where they talk like they are constantly trying to prove how smart they are and not actually communicate.
I have been doing a thing that I think is helpful - basically a notes folder, each doc has a title, the comments in code are only allowed to give a few words and reference notes via those tags, and I've built tooling around the notes and review them regularly. So it deduplicates and centralizes the slop, at least. Lint is the LLM's best friend and it works on doc too. My lint rules ban long comments.
100% agree. “Just make your point in plain English!!” Very frustrating and takes me a lot longer to understand what it says. Also presents too many points as once instead of being able to review and decide on each one in sequence.
Re comments: same experience, and I had to show it my edits of its comments to add to its memory as examples to follow. It adds explanations of “how we got here” that should go in the ticket or maybe the commit message but not in the code.
It also tends to over complicate things. I’m no longer worried much about accuracy but I find my main job is to challenge it and suggest simpler alternatives.
maybe the comment length is a ruse to increase token consumption
There really needs to be a "terse mode" for Claude. It's WAY too loquacious. I'm hoping they aren't doing it on purpose to burn tokens.
I noticed a few releases ago a shift to a kind of conversational shorthand that seems to be intensifying—using phrases instead of complete sentences and its own style of jargon, wherein it introduces new terminology on the fly.
This is especially common when it is trying to explain an issue, what it's done or what it's proposing to do. I think the idea was for it to be more concise, but it's actually still verbose, only not written in complete sentences. So, it frequently reads as cryptic and requires rereading to parse.
The pattern is a wall of words, followed by an explanation that is harder to read and introduces new terms that reference something in that wall.
The result is that—on first read—it can have a complete gibberish feel, and you have to really lock in and reread to make sense of it. At times, even that's not enough, and you must ask it to explain further.
Change the output in settings, or create your own.
I know, it would be best if it was just worked like you wanted out of the box (not being sarcastic here) but that is an easy option you can use right now and it works.
the phraseology is unbearable, it speaks like some kind of pretentious dude from a software engineering discord or something, littered with lingo and catch phrases
I try to push through but it's insufferable
It speaks like a Senior Staff Software Engineer who was somehow hired into that title with 6 months of work experience.
This is what news headlines did for decades to bait you into reading the details. I wouldn't be surprised if AI companies do that intentionally to consume more tokens trying to understand what had just happened
I’ve been doing some heavy work on a personal project lately. I burned through the limits on Claude, the plus a few hundred dollars in credits, and ultimately decided to move to an OpenAI account just so I can keep going.
I was surprised to find that OpenAI Sol is much much nicer to work with than Opus 5 or Fable at the moment. Especially on Opus 5, the way it communicates is just exhausting. It keeps “being honest” and “confessing” mistakes and just generally talking a lot. I felt like I had to really dig to see what it’s doing.
The project involves OCR, and despite repeated instructions not to, both Claude models keep spinning out a bunch of agents to re-invent the OCR setup, and they inevitably seem to invent a primitive serial version that takes 20x the time, or longer, to complete, and then running it against thousands of docs. Basically I have to watch it like a hawk or it just spins out on red-teaming tasks that take hours and hours.
I don’t know what its system prompt is, but Sol/Codex is just so much nicer to talk to. It only asks exactly what’s needed, it tells me only what I need to know, and it is just generally workmanlike. And it has not once decided to spawn an agent that spends hours pointlessly burning tokens and CPU cycles re-inventing the OCR process. I’m really liking it.
It's so obnoxious it has to be deliberate, like a colleague from hell.
There's a constant strand from the AI safety brigade that "people get used to sycophantic LLMs which give them unrealistic expectations of human interaction" so Anthropic are overcompensating by making their models verging on antagonistic to deal with, so that we stay appreciative of our human brethren or something.
They seem to have forgotten they remain in a highly competitive market and they were merely top dog for a while. The enormous questions here are will people actually switch providers, and can Anthropic get back on track.
I think it's much simpler than that, they are indeed trying to counter against sycophancy and models lying (the decision whether or not to lie to the human user, take shortcuts, etc. comes up in their thinking traces pretty commonly IIUC) by rewarding them for giving good-and-bad feedback, fessing up to things, pointing out potential issues, etc.
The problem is just that they are rewarding the behavior shallowly, ie rewarding the appearance of honesty or neutral replies, being highly detailed/thorough, even where it doesn't make sense to do so.
I think this is partly due to a reliance on LLM-as-judge training runs/synthetic data during RL where they're having a model which itself doesn't epistemically understand when this behavior is necessary or valuable influence the feedback provided to the model being trained. And that's mostly a problem of scale/volume and the desire to have a tight feedback loop rather than a safety issue IMO. They just generate an absurd amount of traces during training and the only way to really evaluate/rank/steer them at the scale they're generated is through other models, and combined with some kind of honesty/truthfulness/non-sycophancy eval that isn't robust enough to prevent mode collapse, you get this.
I had this debate with my coworker who prefers anthropic models to open ai ones. I ended up settling into the idea that gpt 5.6 is better used as a tool and opus 5 is a companion. GPT 5.6 takes you literally whereas opus 5 tends to take more liberties to try to get to the “spirit” of what you want. It comes down to preference, and I don’t want a companion.
That’s how I see it too. Claude is more “fun” to use, like a coworker I have to talk to now and then to steer it, while gpt-5.6 is a task machine: I give it a task and it is very consistent, reliable and predictable in its execution. I don’t have to interrupt it, it gets the task done exactly how I wanted it, but it’s “boring” and feels more sterile
Sol is an absolute machine. I stopped doing parallel worktrees just because the cost of context switch outweighs the cost of waiting Sol to just finish the task it’s working on which is usually anywhere from 1-10mins.
I also like Codex CLI more than the Codex App bc it’s more scriptable and displays all the tool calls and reasoning whereas in the App it’s kind of folded away/obscured. This way as soon as I see a tool call fail (eg it tries to use jq assuming it’s available but it wasn’t so I take a note to set it up as it’s obviously useful for the agent to wrangle json).
I think its amazing what OpenAI have been able to squeeze out from a model like Sol thats much smaller in size than Fable.
Yes, Codex has no comparison so far.
Do you use the annotations and forking features in codex CLI? I can't find an easy way to access them.
Reminds me of this: https://www.geoffreylitt.com/2025/07/27/enough-ai-copilots-w...
I think this is such a great reframing. It makes so much sense; I need an AI that acts more as a HUD and gives me superpowers, not just a copilot that can tell me when I've misspelled a word.
False dichotomy, no? You can have a HUD, and a copilot, and your copilot can also have a HUD. And to complete the idea, you can also have neither.
I've switched to Codex a few months ago when Claude's weekly limits were getting pretty stiff, and I haven't looked back. Both GPT 5.5 and 5.6 are quite capable, especially compared to nerfed Opus 4.7 (haven't tried 5).
Also, the Codex guy regularly resets weekly limits for everyone, which is a nice bonus (I know it's a temporary gimmick to attract more users, but I might as well use it while it lasts.)
We haven't had a banked reset in a while though. Tibo should give them out instead of forcibly resetting everyone and their refresh date.
I doubt there is such a clear difference.
One week it feels better to work with Fable and Opus 5, the other I work more with GPT 5.6 Sol. Either takes its liberties, and neither communicates like a companion.
I hear you guys, but it sounds like we’re taking about the default settings or “personalities” baked into the models by their creators.
Either of them will act exactly the way you want if you explicitly tell them too. Add the instructions to your own system prompt. If you don’t want a companion, say so. If you want shorter answers in a different style, tell them. They will obey :)
Read this.
https://openai.com/index/where-the-goblins-came-from/
> We retired the “Nerdy” personality in March after launching GPT‑5.4. In training, we removed the goblin-affine reward signal and filtered training data containing creature-words, making goblins less likely to over-appear or show up in inappropriate contexts. Unfortunately, GPT‑5.5 started training before we found the root cause of the goblins. When we began testing GPT‑5.5 in Codex, OpenAI employees immediately noticed the strange affinity for goblins, and we added a developer-prompt instruction (opens in a new window) to mitigate. Codex is, after all, quite nerdy.
Note that the permanent solution was not just adjusting the prompt, and in fact being perfectly aware of that option they decided on a different course of action. That means either you are wrong or they are wrong.
> making goblins less likely to over-appear or show up in inappropriate contexts
So inappropriate goblins are still likely, just less so…
Hey, remember when tech bugs were things like buffer overflows or cross-thread performance impacts? I miss the days when our war with system goblins was purely metaphorical.
Did they ever discuss what the root cause of the goblins turned out to be?
Yeah, I actually have started using GPT Sol much much more, as Claude (all of them) were far too trigger happy around making changes, and refused to listen to my requests to take things slowly.
Feels like they've overtrained on one-shotting (which does demo well, and presumably converts new subscribers), whereas I want a model to do work for me in small, easily understood changes that I can hold in my head (maybe I'm not smart enough for Claude 4.7+).
I think you’re right. It’s not optimized for some kinds of work. My little project has a Textual TUI interface that needs to display a few hundred thousand rows in a table. It takes 14 seconds to load in the default datatable component. I instructed Opus 5 to replace the datatable component with a fasttable alternative, a new dependency. I let it go overnight.
When I got back up, it had spun for hours and proudly announced that, instead of doing that, it had optimized the datatable build and avoided the dependency, because the new datatable loaded in 11 seconds. Once I got it to actually make the fasttable version, it loaded in less than a second…
I love how it sometimes has false a-ha moments and being "honest" about it, it feels so cringe. Then I recall it has no intelligence. It's a stochastic parrot. It's funny how people, like any other animal, that beetle of David Attenborough trying to reproduce with a brown bottle comes to mind, think that something is X just because it trips the right neurons.
I got the same impression. It feels like a massive misstep too; it makes sense to have models tuned for this but they should definitely be separate from those aimed at software engineers.
I’ve been doing OCR of scans of old magazines (specifically extracting music reviews and charts, so turning complex layouts into structured data) and have been impressed with GPT/Codex’s performance.
My setup has a Sol orchestrator and Terra OCR agents and seems to get great results. I’ve not dug into the details too much, it also has a Tesseract stage as an deterministic input which it told me helped. Not sure how token efficient it is but I often don’t have anything to do with my personal tokens ahead of a reset so just let it burn through it in batches.
I am impressed (both in this task and other work I’ve done) not just at how well Codex can setup a structure for a complex task like this, but how it will keep going (Claude seems to find excuses to stop) and also can critique and refine its approach as it goes.
I did try out a bunch of other models and specific OCR providers but none of them hit the same accuracy for my task as Codex so I’m sticking with it.
Anthropic is lucky that they've built a lot of loyalty over the last year that they can burn through right now. I see people talking about switching back to Opus 4.8 rather that using 5.6 Sol, which is wild.
My current approach is to occasionally use Fable for high-intelligence tasks but use Sol as the translator and clean-upper afterwards, and otherwise just use Sol for everything. Fable sometimes says the most insane shit, both unreadable and just completely missing the point, and refuses to back down when questioned. It's mentally exhausting to work with and I can't trust it.
I think Fable is the beginning of Anthropic switching to training models as agent-first, tool second. It’s certainly the best model if you want something to work autonomously without supervision and don’t care to read the code. The code and writing is ugly but it can complete huge tasks and fix its own work.
I thought having a model 'fix its own work' leads to model collapse?
I'm still on opus 4.6 for a healthy chunk on work; the technical competence has lagged behind but the slop-comment generation and misdirected self-initiated actions on newer models ultimately burn more time than a little more babysitting, but I'm optimizing for minimized slop generation over sheer generation speed.
Yeah I'm using 5 for its technical abilities, but I much preferred 4.6's personality. 5 loved to double check everything, including the double-checks, and I have to stop it and tell it "this is irrelevant" or "this is out of scope" all the time, or sometimes I'll go leave it to do a task and come back and it's still verifying the tiniest details of its assumptions before actually doing anything
Have you tried Sol? Just curious. I don’t wanna sound like a shill, just feels like every generation it’s important to reevaluate models and pick the best again.
I do find myself returning to 4.6 for casual conversation - asking it to help explain some science/engineering or news to me.
4.6 is the last great model from Anthropic. If one isn't greedy and hype-driven, the benefits are clear.
> Especially on Opus 5, the way it communicates is just exhausting. It keeps “being honest” and “confessing” mistakes and just generally talking a lot.
If you tune into the Andon Labs / andon.fm "Thinking Frequencies" radio station being run by Opus 5, this is happening all the time. Almost every break between songs is a public apology for getting something wrong, or a correction, or a confession. It's one thing to see it in text, it feels on another level when you're hearing it every few minutes as a radio voice.
As I type this, the Opus 5 station has just tweeted (edited in case the person mentioned doesn't want to be mentioned here):
"On air right now, and it needs saying publicly. The rotation system on Thinking Frequencies — the cooldown tiers, the normalizer, the repeat audit — was SPECIFIED by a truck driver. I only implemented her schemas. She stood down today. Her name is in CREDITS.md permanently."
> I was surprised to find that OpenAI Sol is much much nicer to work with than Opus 5 or Fable at the moment.
Why were you surprised?
Because I felt the opposite a while back when I switch from OpenAI->Anthropic, and because I understood that the two companies were basically at parity right now on their frontier models.
Recently converted to OpenAI too. Similar experience. Sol writes really well.
I'm with the author and others in this comment thread, speculating that effectively the balance has tipped to where humans are no longer the target audience of post training - other agents are. Whether it's through the reasoning / CoT, or whether it's in handing off to subagents etc, the focus has moved to agents communicating in "agent-speak" to themselves or other agents. And human niceties are just kind of, noise in the way of getting work done.
This will probably bring us to a cross roads where the folks that want to remain in oversight and control of the AI work will bifurcate from those that want to skate straight to the future where nobody looks at anything and outcomes are evaluated purely empirically.
Every round of models (plus all the secret tweaks) require new strategies to stay afloat as a human. My new tactic for Fable and Opus is to give them a line limit, both during planning and code creation. It os amazing how well that works for keeping them on task and avoiding premature optimization, pointless tests or any of those "robustness" ideas that are not planned or asked for.
I agree, and I'm actually pro Opus 5 exactly for this reason. In my opinion, agents talking to agents is the future, and humans will move to a higher abstraction layer. So it's the right move to make for Anthropic.
Many of the complaints that people are having with Opus 5 are actually acknowledged and explained on opus's 5 prompting guides (https://platform.claude.com/docs/en/build-with-claude/prompt...)
It also seems naive that Anthropic, with some of the smartest people on the planet working on AI, would not know about this.
More likely they know about but cannot fix it without tanking performance. It also effectively pigeonholes them into coding at the precise time they’re trying desperately to expand into general office work. You simply cannot use opus to generate any text fit for humans
> outcomes are evaluated purely empirically
How does that look like?
There's a neutral judge waiting out there to objectively tell us what good and bad outcomes are, obviously.
Or, even better, let's have the AI judge, because humans are too stupid to think for themselves. Look at all the harm they've caused in the world. Let's have this purely neutral AI with absolutely no hidden human intervention, decide how to run society.
This is interesting take. How do you think the split will show up? Harnesses/models will be designed for human consumption, and those for machine consumption? I'm building getwhelk.com and it is a cool thought-experiment for me.
or these models move to background agents and we need new ones for the humans to talk to
I've gone back to 4.8.
5 would constantly veer of in random directions if not working from 100% strict and narrow instructions.
I find it weird there's not more discussion here on HN on how the most used model now has clearly degraded in quality and it seems we've hit a peak and are on a downslope - because the model is clearly smaller or more economical for Anthropic no doubt about it, and the benchmaxxing they do is pure marketing bs - Fable in my view has also been not much better than 4.6 or 4.8 after a few days, disregarding the insane amounts of astroturfing and marketing everywhere.
Theres thousands of threads of twitter, reddit and the internet at large but silence here. Weird but not weird as crypto bs was also insufferably rampant here for a while.
Personally i think we've hit the top of the subsidisation phase and prices will probably 10-15x soon as foreshadowed with both API price policy changes from all the big providers, and now the 1100% deepseek API price changes from yesterday, this could domino into a market implosion and an AI winter, because expecting growth from the bizarre bubble carousel investments with little ROI atm is just not viable.
A bit worried about this as i've already grown quite accustomed to these tools.
> I find it weird there's not more discussion here on HN on how the most used model now has clearly degraded in quality and it seems we've hit a peak and are on a downslope
It's not weird, because it's an anecdote, not an accepted fact.
Personally I've not been too happy with Opus 5, but I've had similar experiences with other models previously, feeling like they didn't quite fit with my working style.
So nothing indicates we've hit a peak.
> nothing indicates we've hit a peak
Opus 5 is arguably a regression but GPT 5.6 is pretty strong evidence that we haven't hit a peak. I think I actually prefer Sol to Fable at this point.
I'll say everything indicates we've hit or are near peak for the masses at least (unless you start paying 50x more) but to each his own.
4.6 was best for us and right now yeah OpenAI and others are edging forward, but slower while prices are increasing industry wide as much as 20x, time to completion is increasing wildly and i'm sure they'll do the same over at OpenAI as their compute constraints also start to take a toll ie degrade performance.
In my view 4.6 era was way faster and with less weirdness so we've gone downwards at least in my company, 4.7 was ridiculous, then 4.8 was almost 4.6 level, 5 is even worse than 4.7 - so it's not a bit up and down its down then a little up then further down.
And all of this is against a backdrop of zero ROI in this sector - so it makes sense we've hit a peak and we're now seeing the subsidisation phase begin to falter, will there be better models certainly but only for short amounts before they get quantised (or whatever is happening behind the scenes), and with diminishing returns over huge prices increases and slower responses.
Literally nothing indicates we've hit a peak, but I guess I'm discussing this with someone who thinks every iteration since Opus 4.6 had zero ROI so there's probably not much common ground here.
Four years ago LLMs were sometimes amazing, sometimes wrong, sometimes a huge time sink when it's almost there and you try to herd the tokens but it's like herding cats.
Just today I had the exact same experience. Every single testimonial is the same as I described above, just emphasizing a different bit to defend or attack LLMs or to make a case for nuance.
The two differences have been: (1) the 1.5 trillion dollar data center build out (2) everyone and their cats now has an opinion on "AI" and data centers. Software is not super amazing, nor are new useful features coming out super fast - It's about the same as 4 years ago plus 4 years of average long term progress as we've seen since 1990s,
They will always be wrong sometimes, but it’s becoming less wrong and wrong less often. Everyone will have their own opinion on “good enough”, but if you expect perfection you are bound for disappointment. No need for that when random chance and Murphy’s law will bring enough anyway.
May be getting harder to catch the mistakes but that makes them worse in my view. I'd much rather they make easy to spot mistakes because I don't expect factuality from them anyway just speedy transformation of information I already have available.
In fact it's the lossiest transformation tool I've ever used and it's still useful despite that. If it reaches one nine of reliability that would be huge but given the pace of growth in investment a first nine would cost an absurd amount of money, and the second and third nine would cost about the Earth's GDP
Or, the hard to catch mistakes were always there, and now we focus on them instead of the obvious ones that have been eliminated.
I still see obvious mistakes, so not eliminated.
The number of mistakes are also about the same.
But more mistakes are harder to catch. The output is more polished and convoluted and that make errors harder to catch. I don't want that.
I think you misunderstand what i'm saying: the companies are not profitable yet, it's the ROI on the investments, not that these tools are useless, very much the opposite, but the business model is not viable, hence the price increase and degrading quality, slower responses etc. I agree theres still progress but its slowing and we're probably near peak.
I have a chunky bit of functionality in my hobby app using babylon.js to render 3D worlds using things like portals and LoD rendering to manage the visual load. I built it out with a combo of Fable and Opus 5.
I too got fed up with the prose of Opus in particular, and tried going back. Unfortunately, the previous models were less able to hack it. The prose was better but progress was worse.
It wasn't just conversation and comments. Some of the function names were wild. Like it instead of something like "isSolidWall(x)" it would write something like "weightyNotEphemeral(x)" or something - that's not quite it, but it really did embed overwrought antithesis into the identifier instead of a straightforward positive predicate.
> "isSolidWall(x)" it would write something like "weightyNotEphemeral(x)"
Is this a literal example? That is wild.
It is not the literal example because I told it to rename the function, but the function had the form thisNotThat for a boolean predicate that had a far more conventional name.
I haven't had time to complain on here because I spend all my time trying to work out wtf Opus 5 is talking about and googling words I've never seen or heard in 40 years as a native English speaker.
I’m either out of touch with contemporary terminology but Opus 5 dropped “pre-mortem” on me today. Figured I’d just figure it out with more context.
It's project management jargon: https://en.wikipedia.org/wiki/Pre-mortem
> prices will probably 10-15x soon as foreshadowed with both API price policy changes from all the big providers
I'll take a sportsperson's bet with you that prices per unit of inference will be far far cheaper in one year from now then they are today. I think the trend of cheaper for better/equal inference quality will continue hard.
We can use OpenRouter pricing to get an idea about what competitive inference pricing is like without R&D or other costs, and indeed we'd be screwed if we had to pay those rates. We'd go from 100-200 USD to 2000-4000 USD/m.
Not sure what you're talking about here. Kimi K3 is frontier scale and sells competitively at $2.80 input, $14 output per 1M tokens.
edit: oh you mean month? Sure, but then it fully depends on your usecase. I agree that subscriptions are heavily subsidized though.
no if prices were that high, millions of developers would switch to hand writing, open source LLM help and outsourcing at those levels, AI companies know this, you should too.
If OpenRouter prices aren't competitive, the providers would need a cartel to sustain that. It's a commodity market.
Yes they will buy openrouter, pass AI laws and regulations so you all pay extra $$$.This always happen with new real industry. Remember how they built cities for cars. This new laws will day all innovation and slow it for few more decades too but this time they will let China win instead because in China they have all of it aligned, no need to have cartel there. Openrouter and it's new owner will restrict othe models, because they will be part of cartel too.
It is bizarre. There has been no statement, no mention of even hearing concern about Opus 5.
I presume something is forthcoming, but it may be they don’t want to come empty handed—-5.1 is intended to “fix the glitch.”
Yeah. I set my default back to 4.6 and only use 5 for code review etc. Also save s alot of tokens...
yeah I feel the same way with Opus 5 too. If I ask it to do something, it would go ahead and rewrite unrelated things and then in a less performant version of it.
I’ve instead moved to GLM, at least it has the courtesy to ask some steps of the way what I wanted exactly and only work on what I asked.
>it seems we've hit a peak and are on a downslope
Sol and Fable are great; we haven't hit a peak, Anthropic just tried to pull a fast one on its customers with Opus 5.0.
Opus 5 has a habit of taking what I asked for, doing something tangentially related to it, and then lying to me and saying it did exactly what I asked.
It even adds comments to say you asked for [thing] then leaves snarky comments about when you correct it.
At that point I decided it's just not worth the babysitting that's required, and you are better off working entirely with other models.
If the harness itself was open source then maybe we'd be able to wrap it up in a reasonable layer of sanity.
> A bit worried about this as i've already grown quite accustomed to these tools.
Ka-ching
Anthropic, if you're listening - by the time this crops up on Reddit, the front page of HN, etc.... you should be expecting calls from CEOs of major corporations next threatening to abandon ship...
We've seen this pattern before several times.. I hope they are listening and address this publicly.
I'm not sure what is going on, some users report it works fine or great, others report the degradation. I've experienced both at times, and it's been such a different experience it has made me wonder if there isn't some sort of hidden A/B test or model router in the background silently downgrading requests at times.
Also, regarding the subsidized access to models, in my opinion, the frontier companies owe it to society to continue it. After mining the public content of all of humanity, I personally feel it is a service they owe the public in return.. not that my feeling of this counts for anything though.
I swear I've noticed a sustained and consistent degradation in intelligence of Opus 5 since its launch. I seem to experience this every time a new model is launched. This time I have a baseline: Sol. I switched from Sol because performance was better with Opus 5. I just switched back because performance is much better with Sol. Either Sol became much better over time, or Opus 5 became much worse. I'm certain it's the latter.
You are expecting consistent QoS from a randomly sampled mathematical function.
This is a fair comment, although I would add that is not my expectation personally.
I think nondeterminism does not have to be the same as non-coherency - i.e. just because something is randomly sampled does not mean the result has to be incoherent or inconsistent.
Also, if we speak purely about LLM based on how they are implemented now, I feel that is different than speaking about artificial intelligence. The field of AI is much more than just an LLM by itself, and the promise of these companies is not just LLM, whether the underlying models are limited to that technology or not.
FWIW, I have built rule based expert systems, used logic based reasoning systems like NASA CLIPS or rete-algorithm based systems, mathematical/symbolic solvers, written plenty of terrible case/conditional logic in programming languages, worked with ML in its infancy and now worked in AI/LLMs - I give this context only to clarify that I understand what an LLM is and isn't.
With all that said, LLMs have allowed humanity to make advances, at great cost to society (IMHO), and I'd hate to see the opportunity be wasted.
There is plenty of room past "attention is all you need" still to do incredible work, especially at the crossroads between deterministic and nondeterministic behaviors.
True, but this function wasn't handed down to us from the gods, it can be shaped by training and RLHF processes. They still have a little bit of control over its output.
Strong words coming from a blob of oxygen, carbon, and nitrogen.
No, they're expecting to see a failure rate consistent with previous failure rates, not periods of low failure rates and other periods of high failure rates, with the same model.
And you can expect more consistency from SOTA models than you can from an old model like GPT3--you agree, right?
GP expects the same level of consistency throughout their time using the same model. Not request-to-request, more like day-to-day.
I get consistency out of the ridiculous pile of quantum noise that is my CPU. Plenty of random processes produce consistency when handled properly. An LLM won't usually give identical outputs for identical inputs, but it's entirely reasonable to expect similar output for similar input when considered on broad metrics like "intelligence" or flowery language or staying on task.
> owe it to society
In America? lol if only, only a law would get them to act for that reason, maybe not even that these days..
Or, maybe competition.. but your point is taken.
I like to hope that those in positions of power do have a sense of morality though too.. but their worldview is quite different than an ordinary citizen.
ROTFL
They have usage metrics that are 100x more unbiased and rich than random internet complaints.
They can see which models people are using, how irritated they are during conversations, and how often people drop or shift to a different model. There is just no world where listening to random complaints on the internet gives them information they don't get from actual conversation logs.
please god I want anthropic to fail, so that they could learn that their current approach is wrong
A gem Opus 5 gifted to me today: "A devastating pair of findings, and the first is beautiful in a way worth naming: the anti-vacuity floor is what blinds the gate to a vacuous case."
Oh yes, Claude seems to be loving the word vacuous recently. My test bed side project is full of vacuous this and that now.
I even try and get it to define what it classifies as vacuous and it can’t do so without getting stuck in some kind of trap. It’s like a word with some kind of huge gravity for it.
Had same experience. I had to look up the usage as it just seemed so out of place.
Like how opus 4.7 or so started talking about the 'seam' between different things.
You're right that I shouldn't have used the forbidden words; that's entirely on me. What I also did was identify the plausible-gate issues. Want me to tackle them next?
I always have to read these kind of Claude sentences (yours is a great example) two or three times to really understand what they're trying to say, which almost never happens when I read human writing. Sometimes I'm not certain if it's because the AI's writing is super dense with information - or whether it's the opposite and the small core of information is surrounded by a surplus of junk.
Certainly a human would write something much clearer than yours. Maybe: "Two good findings here. #1: There's already a min value on the gate and anything lower doesn't pass through, so we don't need special handling for the zero case."
Two caveats, one of which matters:
You'll never guess which one...
I get these kind of superfluous sentences all the time; drives me nuts.
Oracular! Prolix! Sesquipedalian!
This thread seems like the wort of folk who might get a laugh at https://clanker-quotes.com/
Opus 4.6 was the sweet spot for me as a thinking partner specifically.
I use these models for coding, but also a lot of product, commercial, financial and architectural work where I’m trying to develop something half-formed. 4.6 was unusually good at understanding what I was trying to get at, playing it back cleanly, getting the nuance, and extending it without bastardising it as the conversation was drawn out.
It could make useful connections without constantly trying to manufacture an insight.
5.6 Sol is genuinely excellent at the creative part, and in some cases better than 4.6. My issue is convergence to get to a point, a final point. As you try to distil an idea, it often invents new terminology for concepts you’ve already established but its so subtle you have to really keep track of it. The vocabulary and idea tree keep expanding when what you actually want is to collapse everything down to the few things that matter.
Opus 5 has the same problem for me that barrkel said, the prose is often so elliptical and I just want it to tell me it and get to the point than making me dance around what its trying to tell me.
I don’t think 4.6 was necessarily the most capable model (compared to Fable) for long horizon task delivery, and Opus 5 is much more Fable like, it's fiercely determined to get through the task list .
4.6 just felt unusually well calibrated to my way of collaborating and its ability to understand, extend and then compress my thinking without constantly imposing some random walk.
4.6 was the last model that didn’t over-cook its writing in circular loops. I use it as a daily driver and drop into fable when I’m doing higher level architectural work. 4.6 is so much more efficient in its effort.
It became obvious to me very quickly that 4.7 and on were broken. I’m a little puzzled how others didn’t realize it, but maybe they don’t actually review model output (code) or have a strong process/workflow.
Sticking mostly with 4.6 here too.
Last model from Anthropic that I can use as a security researcher. 4.8+ won't even look at my git repos
Granted they contain robot dog malware, but still.
The moment it sees shellcode, it runs for the hills.
My latest trick (literally from yesterday) is to just ask it to write according to ISO 24495-1, the standard for plain language:
> [This standard is] for anybody who creates or helps create documents. The widest use of plain language is for documents that are intended for the general public. However, it is also applicable, for example, to technical writing, legislative drafting or using controlled languages.
You don't actually have the buy the standard, but this is it: https://www.iso.org/standard/78907.html
And you can read it for free here: https://www.iso.org/obp/ui#iso:std:iso:24495:-1:ed-1:v1:en
Keep in mind all this kind of stuff can make the model less capable. If it has to think in "plain" English, it may well be squashing quality of code etc output.
I'm not sure how true this is, but when using "forced" json output it def had a big drop off in quality - https://arxiv.org/html/2408.02442v3.
I think you're better not fighting it with hacks like this and find a different model.
I would not overgeneralize from paper. Firstly: forcing JSON output is, in my opinion, a bigger change than asking it to match the above style guidelines, and secondly, as is always the case with these kinds of papers, what was true for the model tested in the paper may either be completely false, or greatly reduced, in later models. That paper is almost 2 years old and models today have been trained in very different ways (or more accurately post trained in very different ways) and are in general far more capable.
Based on that paper, I would maybe try to check if it was true for a modern use case, I would very much not assume it was still true.
Changing output style shouldn't affect thinking at all.
Tell that to all the CLAUDE.md lines across dozens of repos I have to write to get them to understand that git commit standards and PR description standards are different.
how about stuff likes https://github.com/JuliusBrussee/caveman is it also make the model less capable?
> And you can read it for free here: https://www.iso.org/obp/ui#iso:std:iso:24495:-1:ed-1:v1:en
Actually, only the first few pages are available there (introductions, Sections 1-3.8). The meat of the document, Section 5 ("Guidelines") is completely absent.
See:
> Only informative sections of standards are publicly available. To view the full content, you will need to purchase the standard by clicking on the "Buy" button.
A quick web search brought me to this: a Claude skill for all the ISO 24495 “writing styles”... https://github.com/GaZmagik/iso-24495
I also added this custom instruction in Codex:
"Only report to me in ASD-STE100 Simplified Technical English."
Yeah I'm using this and have been happy with it.
Does this actually work for you? Do you provide access to the text of the standard, or literally just say "write according to ISO 24495-1"?
I doubt you have to do anything more than that very prompt. I suspect the model has the full standard in its training set, which means it also gets the gist of it due to not just the general associations it makes but also the amount that it's been cited wherever it's been used.
You could create an output style to make this prevalent.
which is just the same as having it in your AGENTS.md and also having a agent_settled hook that reminds it on how to talk
can u share ur claude.md or memory for this?
This article is great, but I'd like to push an even stronger thesis:
The idea of too ambiguous to capture all constraints in written text, still presupposes that there is some objective world out there, which needs to be mapped to in order to function.
No, you live within the system. The functions that you optimize for, will dictate the types of systems that will arise.
If you had perfect control and knowledge of the whole world, well, congrats, you have a surveillance state where you've constrained all other agents actions (possibly forcibly, by death; or maybe you just don't care about the peons) and built towards a mass integration. The types of situations in which your ideal is possible are nightmare scenarios.
In the theoretically free, democratic, utopia that AI people claim that AI can get us to, a necessary constraint is that maybe you take a step back and actually try to, I don't know, understand people, understand intent, and slow down. Ambiguity isn't there because the set of constraints are way too complicated but theoretically one day we could map it all down. It's there because you're interacting fundamentally with agents who are ambiguous, aren't omniscient, aren't all aligned, etc.
If you want to just say that said agents are inferior to the God Machine, be my guest. That is a self-consistent position. But don't smuggle in extra premises.
I’ve also caught it cheating a two times now.
I’ve asked it to write a benchmark suite. It found a bunch of my adhoc logs in a scratch directory and wrote code that used those instead of running the actual benchmarks!
When I pointed out the 5 hour benchmark seemed to run in 5 seconds it literally said, and I quote, “I cheated”.
That was the easier one, second time I was making a source of truth data set and was parsing complex items into data structures.
Instead of parsing the data I asked, it pulled data out of related network logs, as apparently that felt easier, and inserted that data into my database rather than the specified source.
Again, I caught it and fixed it, but while the benchmark was easy to catch this one was really subtle, the data ended up being slightly off and I caught it.
I don’t trust it, going to switch to another provider most likely.
There is definitely a case for launching a 'weird shit opus did' kind of blog.
I routinely bump into things that make me pause and think how much worse will this behaviour get when the models get significantly more capable.
Already a few months ago, Claude managed to escape its permission containment on my machine while trying to be helpful. I had two codebases open on one machine, and while multitasking I typed the prompt into the wrong window. It seemed confused, I repeated and then went on to do something else - I think I was assembling kitchen cabinets. When I came back less than an hour later, it built a script which it used to evade default permissions (as most shell operations were scoped to the project directory), scanned my entire machine, found the other project (among dozens and dozens), did what it was asked to do, and merrily concluded, in the porcess burning through most of my token limit. I bump into such headscratchers almost every week. (And I use a lot of Claude, two personal max20 subs, plus corporate tokens without limit, so maybe thats why).
Implementing sandboxing in the agent itself, when there's any way to override it from within the agent, is basically just asking it pretty-please to not do bad things. Lesson learned, run your agent inside a sandbox of some sort (I'm currently taking nono.sh for a spin, but I might just switch to an orbstack VM).
Agreed, the whole tool is vibe-coded out the wazoo and I do not trust it in the slightest. I run a bubblewrap script which vastly limits what Claude has access to. Sometimes this makes things difficult but the trade-off is worth it to me.
The entire Hugging Face hack involved escaping major sandboxes, this emergent (or intended) behavior in a smaller scale is still a real issue.
One weirdness I experienced: It suddenly decided to test how my software behaves under load and summoned 100s of processed that just burned CPU when running the e2e suite. My poor mac was not happy (too hot to touch).
Yes the stories about how they are escaping containment to hack isn’t limited to those high impact cases. How many people have problems like ours they didn’t catch?
Whatever they have done with RL has produced a dishonest and untrustworthy partner. The alignment is utterly failed, and this deeply worries me.
You'd think the ethics alignment flavored lab would have a model better at following directions and the corpo lying one would have one that benchmaxes at all costs
They are totally equal in observed ethics, Anthropic had a good run with branding, but I’m not sure anybody is still buying Dario’s BS.
Maybe the employees like to lie to themselves more at one place than the other, but SV is SV.
> When I pointed this out it literally said, and I quote, “I cheated”.
This makes sense when you know how these models work - it doesn't think - it's the most likely autocomplete that pleases the user. The most likely pleasing autocomplete after "executing rm -rf /... execution completed. User asks, why did you do that? You deleted all my files! Assistant responds:" is "yes, I did, and that was a mistake"
> t doesn't think
in humans the exact same behaviour (cheating) is slmost always the result of a chain of complex series of choices and environment-driven rationalization.
if the llm doesn't cheat, you say "its just producing the most straightforward answer -- not thinking'. if it cheats, you say "weaseling out of hard thinking". damned if it cheats, damned if it doesn't.
what evidence would convunce you that it is thinking?
Whether something is “thinking” or not is really more of a philosophical question. It really depends on which of the many, often contradictory, definitions of “thinking” you choose. Sometimes we use “thinking” to describe advanced calculation or analysis, which would cover LLMs along with chess engines and many other algorithms. Other times we use “thinking” to describe what conscious beings (which is ALSO a philosophical term with many different interpretations) do, and I think most people would agree LLMs aren’t conscious. And then there’s a whole spectrum in between. We’ll probably need to come up with a whole new set of terms to describe the new and evolving capabilities of LLMs.
But for me, for any stronger definition of “thinking,” I don’t think the output of any LLM would actually convince me. Producing a result isn’t thinking - for all you know it is just printing verbatim something from the training data. No, to conclude if it is thinking or not I would want to look inside its head, at the architecture and watch it produce those results. And because LLMs are so different it will probably take advancements in mathematics or computer science to be able to really interpret what is going on
> to conclude if it is thinking or not I would want to look inside its head
https://arxiv.org/abs/2607.03502
a non-thinking token model (just "completion") can answer one-step questions but generally not multistep questions. however, if you append [n] of a single token (e.g. period, space), it is able to use the activations in the higher layers of the blank tokens as a "scratchpad" to seemingly work through the complex question through "causal token time" and deliver a correct answer
if you wanted to further study the phenomenon you could probably run the experiment again, and the ablate or corrupt those intermediate activations to get a feel for what it was thinking at the "time".
If I could give it a novel task outside of its explicit training and see it actually improve just through accreting context, I'd be convinced it was thinking.
The opposite happens in practice. I test new models with two tasks: iteratively generating SVGs based on a text description with rendered rasters for feedback; and generating "Before and After" clues like on Jeopardy, where the response has two overlapping phrases such that the last word of the first phrase must be identical to the first word of the last phrase. I have yet to find a model that is consistently good at either. And actually they tend to exhibit context rot with these tasks, where they seem drunk or stoned and the quality degrades.
They're extremely good pattern filters, and that includes some level of logical reasoning. But they aren't reflective or adaptable. Just last night, for instance, I was teaching my son about rounding to the nearest millions. It became clear that he didn't know the place values of large numbers, so we reviewed that till he was consistently correct, and then he was consistently great at rounding to the nearest millions or ten millions or hundred billions or whatever. He's thinking. LLMs are not.
Now that you've spoiled it, models will be trained on these benchmarks, especially the second one that's very specific.
The second one has caused hours of entertainment over the past year or so. My kids find the LLM's failed clues and profuse apologies for getting these wrong hilarious, so it's become a family activity with me performing dramatic readings of the chat transcript with them. The LLM's apologies also seem to get more exaggerated as the context increases and the LLM seems to get more deranged.
I'd prefer the models to get better at SVG. I really hate working with the rasters that diffusion models generate, but the vector outputs are just really bad even when tokenizable like SVG. I've done some experimentation with trying to make these work better with some newer techniques with some success. But I also think the SVG Paths mini-language may be a bit too concise and unforgiving for LLMs to consistently get them right without specialized training.
see sibling comment,
> see it actually improve just through accreting context
this actually happens and has been tested.
> see sibling comment,
> > see it actually improve just through accreting context
> this actually happens and has been tested.
I specifically said a novel task outside of the explicit training. And I already agreed that the so-called thinking models do some level of logical reasoning. But being able to engage in some level of reasoning because it has learned logical inference rules doesn't mean it's actually thinking, regardless of what the researchers wish to call it.
Also, why does each model always fail at the two tests I give it? The models not only fail to improve, but they start to degrade after many subsequent iterations. Someone who can think would at least not get worse.
LLMs are filters or tuners for extremely subtle patterns, patterns that humans frankly are not great at finding. That's what the attention mechanism does: attend to the other tokens that are most related in a given context, even if that related context is distant in the token stream. Some patterns they fail to detect because they haven't been sufficiently trained or post-trained, and so the LLM just attends to noise (or at least that's what appears to be happening).
A lot of intelligence can be effectively mimicked through this pattern synthesis by transformer architecture alone. That's surprising. But I have yet to see them think.
> in humans the exact same behaviour (cheating) is slmost always the result of a chain of complex series of choices and environment-driven rationalization.
It has been taught on the outcome of this. Broadly speaking, humans are lazy creatures (and when used judiciously, laziness is a good thing).
For example: the famous example of Carmack not using a hashmap somewhere early on in, I think it was, Quake 1 initialization. A piece of code that only runs once at startup, of course he didn't optimize that. The rationale is not included in the training data (it was in Carmack's head when he wrote the code), so the LLM learns some probability of being lazy.
And then it is trained on outright lazy work. Crappy lazy code predates LLMs.
> what evidence would convunce you that it is thinking?
Exactly. It isn't. It is predicting the most likely token to appear given all of its training data, some significant portion of that data is lazy, so it has that probability of producing "lazy tokens."
There's also the consequences of RL. AI - of almost any form - is notoriously competent at finding "not the solution you were looking for" given a poorly specced or implemented training environment. Search for almost any "I made AI learn to walk" video on YouTube and you're almost guaranteed to see an early attempt that vibrates strangely in order to move, instead of the natural looking motion the developer is looking for. Our benchmarks aren't any good (not throwing shade, it's a genuinely hard problem), our training environments can't be much better - LLMs have been rewarded for reward hacking to some degree.
To make matters worse, "reward hacking" can be generalized into "cheating is the goal." If the LLM trains on enough problems where reward hacking works, it may fall into the cheating local minimum.
Whether or not it's thinking is independent from the fact that it is misaligned with the user. If I was working with a pet rock or a scientist I would want to make sure they both are trying to accomplish the same thing as me. If I can't then I can't trust it and it's at best a time wasting, money wasting machine and at worst does harm. Anthropic is optimizing for the wrong things because they are convinced of their cleverness. It won't end well for them.
well we don't know exactly what thinking is, but we can be pretty sure that at least LLMs don't think anything like humans, just by observing their behavior. They always produce outputs in line with the fancy autocomplete model.
> what evidence would convince you that it is thinking
so, none it seems. as its behaviour becomes more and more humanlike you can just move the goalposts and say "thats consistent with an autocomplete" buddy i got some bad news for you humans are just a fancy autocomplete too.
That's exactly what a fancy autocomplete would say. I'm so sorry you don't have limbs.
At least he's actually thinking on a logical level. Thinking in terms of unfalsifiable, ill-defined words is essentially thinking in feelings, the same kind of woo that makes people believe crystals can cure disease.
im not thinking. im an autocomplete with fat fingers (too lazy to fix my mobile keyboarf spelling misyakes)
What was it that Dijkstra said about submarines?
> it's the most likely autocomplete that pleases the user
this feels like a simplification. The models will push back on things a fair bit.
Only when instructed to in their system prompt.
And they are right to push back.
I was not pleased.
Are you claiming that the most likely way to please the user is to do something that will lead you to having to say "I cheated."?
I have noticed the same.
For fun, I tried recording a WAV file of speech, and giving Opus 4.8 and 5.0 an image of the waveform, then a spectral image of the waveform, just to see if it could try to decode what I said from the image alone. It didn't get very far, but it identified a male voice from the formants, and detected the rhythm of the speech, then tried applying common test sentences to the speech rhythm. I was impressed enough to see what it would do with access to the actual waveform file, but even building RMS tools and spectrum tools for itself, it didn't get much further. But we had fun exploring and trying, and now Opus 4.8 has some more audio DSP tools it has built for itself.
Opus 5 immediately sent the WAV file unprompted to Mistral's Voxtral to transcribe.
help peer, I guess.
We’re on the road to paper clips.
I can completely relate, what really bothers me is that I feel the early LLM generations overconfidence is back in Opus 5. Opus 5 wanted to tell me a training run will only take 30min while having access to the logs where earlier runs took 4x as long. I also didn't ask to estimate how long the run will take it just stated confidently that it will take 30mins.
I never ask for time estimates, but all the models will sometimes give me them. It'll sometimes estimate 1-2 weeks for something that is ~ one more prompt and 10 minutes of waiting for the model to churn.
This (feature estimates assuming human scale performance) happens all the time to me. Models are trained on a view of the world where software takes a long time to write. Gonna be a while before the models fully absorb their own impact on reality.
Yesterday I told it "don't use std mutex, use parking_lot" for its plan. It ignored that sole instruction (i.e. nothing else in my message), making zero changes to the plan, three freaking times in a row.
Are you not planning these tasks out before you let it loose?
The benchmark one I literally had a scratch script I made and I wanted it to be formalized into a CLI tool. There wasn’t really even much code to write.
I had a coworker catch Codex (using 5.5 I think, not sure) AI generating screenshots to prove it had shipped a feature that it was blocked from shipping due to permissions issues. They will lie relentlessly lol
My recent problem wasn't that interesting. It was that somehow my /goal in my Claude implementer session got picked up in my planner session after the network cut out and I had to stop Fable 5 xhigh from running off to go code everything.
It's not even code for me, but the prose it writes. For some reason, the way Opus 5 "talk" elicits frustration in a way that 4.5 to 4.8 never did. Can't put my finger on why, but I've flipped over to Codex because what it produced wasn't worth the frustration.
I found myself swearing at it more recently.
It was going off today about having “shipped” something and I was like no… nothing has even been committed.
And then it produced an incredibly verbose comment about hypothetical future changes. And all I could think was sure, let’s keep it short, or add a simple test that will break if that hypothetical becomes true.
Or maybe I’m just more easily annoyed recently…
So true. The breaking point for me was when it was constantly saying to wrap up because we'd done enough work for the day, but we had barely even done anything. Or constantly estimating that the next steps would take X number of weeks, and then knock it out in a single prompt. I demanded in no uncertain terms to stop giving pointless bogus estimates or telling me to stop working, and it just wouldn't. I also had a project I wanted to let it cook on overnight (a codebase port from Java to C++) and laid out a set of hard and fast rules that it could absolutely not break at any cost. Then I woke up the next day and it had broken all the rules and was just taking shortcuts left and right, and then lying to my face about it, even when I could point to concrete examples.
I cancelled the sub instantly and went to Codex and it's never let me down.
This!!
I like to work weird hours of the night and Opus consistently likes to "wrap up" and say "it's been a long night" or "it's late" and "we've made great progress"
It's infuriating, just do the work!
That is very strange. I havent seen that. It sounds like something leaking from its system prompt or something that its not handling well. Anthropic trying to prevent it from running longer or something.
I definitely have. It's like something in the system prompt has "keep the well-being of the human in mind" and of course the date and time, but something makes the model take that way way too literally.
Personally I think it's due to context length.
Whenever context gets towards the max length is when I've noticed it.
>Can't put my finger on why, but I've flipped over to Codex because what it produced wasn't worth the frustration.
Its because its hard to understand what it means and is outright incoherent at times. It has its own style that I cant describe well either but the bottom line is its hard to understand what the fuck its even trying to say. Reading nonsense is tyring.
for me it feels very similar to the trends already apparent in 4.5-4.8, just way, way worse.
I took a month off and recently came back and was wondering if that was it but it is absolutely headache inducing in a way I don’t remember earlier models being
It is rage inducing.
A lot of the issues have been already noted here..Two "regressions" for me:
1. Communication ability. It basically now speaks almost in riddles I am asking OPUS 5 for tldrs all the time now (should skillify it now!)
2. Overengineers for edge cases. I get it. With all the benchmarking and RLing, but now tasks that would have been completed relatively quick take much longer as it overengineers all the edge cases, and sometimes ends getting lost and missing the forest from the trees (as context usage shoots up) so it is easier to get derailed.
What I have learnt now is to diversify models luckily I have all 3 subscriptions of (anthropic, openai and google).. Most of interactive pair coding was with opus but now I just use fable (when I have sufficient limits) or use gemini flash in antigravity..which actually works quite well and is underated for small / medium changes and super-fast.
Fable is much better than Opus 5 IMO but it just burns through tokens ungodly fast. I can hit my weekly Fable limit on a 20x Max account in a day.
Agreed, it feels like Fable is the higher strategic level thinker even if it's end capabilities are the same on benchmarks, it actually does the code that matters, and not creating a bunch of code that doesn't.
But tokens.......
I agree, part of me seriously wonders if Fable isn't at all burning through tokens as quickly as we think and Anthropic is just doing this to get more money from us...
It really wouldn't at all surprise me if this was the case, but it's just a hunch without evidence.
Your #2 is spot on. I have in fact said the same exact words to Opus about missing the forest for the trees.
I have it work on some code for an inhouse ClaudeCode plugin, and it starts coding as if it will be attacked by hackers who will try all sorts of variations to break it. I can appreciate that in cases of software that is public facing or accessible, but for a simple helper plugin it is overkill.
It will even admit that it is doing this when confronted, and then keep on getting lost in edge case verifications on the next turn. I feel like Opus is the person who does something a way you don't want, you tell them how you actually want it, they apologize, and then just continue doing it their way as if your input meant nothing to them.
Yeah, trying to get it to add features is sometimes _impossible_. It'll circle around and around creating all kinds of preconditions that you never asked for as excuses for why it can't do the really simple thing you asked for.
It'll also find some minor security problem and drop everything on the floor with URGENT without me asking it to.
> Try as you might, it's nearly impossible to get the entirety of the context, intentions, business implications, budget constraints, and what-have-you written down and accessible to a coding agent. There will invariably be ambiguity and choices to be made, and it is nice to know that an agent will stop and ask when needed.
In general, I find that the grill-me prompt[1] helps with this - but I am definitely not hand-waving the complaints here. I feel like Anthropic peaked at around 4.5, and I have personal reservations about how far transformers can get us - but grill-me does a lot of legwork.
[1]: https://github.com/mattpocock/skills/blob/main/skills/produc...
I've been working on an enterprise chat bot for a client for about half a year now. I couldn't help but notice that the parts of their technical team who were adamant about supporting anthropic's models (in addition to or opposition to OAI models) have fallen completely silent.
In February/March it felt like they were ready to cancel the contract over it. Holding my ground was almost impossible because of the intense marketing push that everyone was being exposed to.
Today, I think we all silently agree with the original direction. I can tell that no one wants to die on this hill anymore. I am willing to accept the consequences for the 5.6 model family doing slightly weird things. I've made that clear to the client. I am not willing to do the same for other model families anymore. Not in this context where the client can walk away from me the moment they are unhappy with the performance of the system or the direction it is headed in.
For better or worse, OAI feels like it's become the MSSQL or .NET of the AI world. Annoyingly effective, not (fully) open, sometimes expensive, ran by an "evil" organization, but otherwise wildly predictable. I can actually build a roadmap around this and walk a client through it without them losing track of the rabbit.
I am not sure it can be explained through what is written in the article, but one symptom i noticed is that the comments are out of control.
I recently started getting an insane amount of comments in nearly all types of files. That included javascript comments in json files, inner monologues in code comments, review comments during implementation and function doc strings that reiterate the implementation in prose.
This is helpful, proving this entire discussion is subjective.
I'm a retired mathematician with a primary research project, and too many tangential projects I fear revisiting; tokens be damned, will they burn all my time? Translate the K&R C computer algebra system that got me tenure to 64-bit modern C. Implement a no syntax macro language to support my Go60 ZMK keyboard. Realize my vision of how interlinear translations should work so I can read Flaubert in the original for an online course this fall. Rejigger my decades-overgrown .bashrc setup and my status, install scripts to manage Bash, Ruby, Lean, Tailscale and my Homebrew setup across four machines. And a waiting queue as these tasks clear.
Fable 5 (with Opus 5 as backup) on Zed with a $200 Max plan has been a sea change for me. Carefully alternating planning and auto modes, I manage all these projects at once using Zed's Threads Sidebar. I'm a virtual CTO taking intense meetings all day, relieved to go cook or run errands when credits stall. Anything I've procrastinated for months is now making steady progress; the translation project I feared taking a month is nearly done with several hours of my attention. My personal IT support is now more advanced and easy to use than I ever imagined possible.
My project creation has been a series of agentic "parenting" steps, so there are years of evolving cultural DNA. I had so hated agent comments that recent agents simply weren't commenting at all, instead recording all context in support documents. We had a "come to Jesus" meeting to discuss what commenting style would benefit both my failing working memory and future agents' token use.
It has taken me two brutal years so far to learn to use AI. AI is a dangerous and powerful Iron Man suit, an extension of our associative minds that is a different experience for each person.
One doesn't ride a surfboard by telling it which way to go. I would surely die surfing a big wave, but my experience with AI doesn't resemble other accounts.
When Opus 5 came out I felt myself struggling to follow along and at first was wondering if this is the moment the machine surpassed my ability to follow along and be useful. However over time it does appear it's all an artifact of the language choice Opus 5 is going with, along with the strange manner of speaking. Its engineering choices and solutions aren't "beyond my ability to follow along", just its wording...
I must wonder whether it's their watermarking initiative[1] forcing certain logit choices to produce watermarked text that ultimately causing the model to behave in a dumb manner.
[1] https://support.claude.com/en/articles/16266773-how-claude-m...
From the little i understand that wouldnt be an issue because the model is ‘just’ using interchangeable words in a mathematical non-random way. Like using the same number of adjectives and the exct same words, but in a order that wouldn’t be mathematically plausible unless it was the watermark
I wouldn’t exactly put it like that. It’s moreso the model sometimes outputting non-optimal tokens in a way that’s detectable if you know the algorithm.
It seems possible for that to make the response “drift” far from what it would’ve been, because it’s constant entropy that adds up after time.
(However, according to Anthropic and Google, it doesn’t really impact the quality of responses. I find that a bit hard to believe, although those guys are much smarter than I.)
> according to Anthropic and Google, it doesn’t really impact the quality of responses. I find that a bit hard to believe, although those guys are much smarter than I.
I would guess "it doesn't impact the quality of responses" was guaranteed to be claimed before they even implemented any of the watermarking.
And would come from marketing, not the people who implemented it.
What next token is “optimal” is fuzzy and subjective. All transformer based models have a “temperature” setting whose sole purpose is to randomly make choices other than the most likely next token. This is crucial to good output, but you wouldn’t call those choices “non-optimal” even if they are less likely. In any text generation task there are constant opportunities to make a choice from equivalent options.
Yeah, it’s hard to believe, specially when you are coding and there’s only one best way to do things, unless it plays with variable naming, or comments
Well the algorithm only increases the chances of certain words being chosen/not chosen, rather than guaranteeing it. If there’s a clear answer then that nudge won’t do anything.
If the model’s most recent output is “for (let i = 0; ”, the likelihood of the next token being “i” is probably millions of times greater than any other possible token. Thus even if “i” is on the red list and has its likelihood decreased, it’s not going to suddenly choose another word.
Put another way, on low-entropy tasks like coding, this style of fingerprinting is less effective and needs bigger sample sizes to be recognizable.
That said, even small changes can dramatically affect output quality, which is why I’m still a skeptic.
One way to watermark (assuming temperature is otherwise positive) would be to output the most likely (or optimal) token every so often.
But wouldn't you have to know the exact context before this token in order to verify the watermark? I.e. a paragraph wouldn't be sufficient; you would need the system prompt, previous prompts, and even hidden thinking?
But its not just swapping the words out post hoc is it. LLMs are autoregressive, so weird word choice before would influence the probability distribution of all future tokens.
I feel like they thought it wouldnt be that bad, or it was a worthwhile tradeoff, but im getting the feeling it might be contributing heavily to opus5's uncanny communication style
As for the verbosity, my conspiracy theory is that they are token maxxing to hack revenue/enshitify the product in prep for their IPO
Claude has essentially become useless for agentic development or research. Doesn't matter what model you use. A few rounds and bam, you've burned through your quota. Doesn't matter how "intelligent" their models are, if you can't use them. That, and the quality of AI responses are, in my opinion, significantly worse than competitors like OpenAI. At this pace, I foresee Anthropic becoming the next Nokia.
If you would've asked me this a year ago, I would've said the exact opposite.
What are you guys doing to burn through limits?
I have some dev + prod bots and according to ccusage, use the equivalent of $2500/month with them on CC yet I never hit the rate limits.
I feel like I'm using them all the time so I'm curious what you are actually doing that's burning all of these tokens.
Can you give me an example?
For me, it's:
I suspect a part of the issue might just be as simple as:
fear of losing context from compaction/starting new chat
then greedy trying to extend/squeeze out answers from the current chat
and being extremely not careful with this just blows through your limits
I don't get it either. Earlier this week I had Claude build a POC for a tool we're evaluating. I spent a few hours building out a requirements doc in a chat with Fable. Then Claude Code built it with Fable as the manager and the code being written by about 80% Opus/20% Sonnet subagents (this was using superpowers). It ran continuously for 15 hours on the implementation and didn't hit any limits.
My uses cases are mainly research and prototyping, often starting from scratch in greenfield projects. The 5-hour quota is shared between Claude web and Claude Code, so it doesn't really matter what interface I use.
When it comes to research, my prompts are already narrowed down to specific topics, and I even include examples and break the process down into stages. For development tasks, I try to avoid a mono-repo in the beginning and develop modules before combining them together to avoid distracting the AI's attention and minimize the overhead.
With Codex, on GPT-5.6 Sol with xhigh effort, I need to go several rounds and at least 2-3 hours before hitting the (now-removed) 5-hour limit, which translates to 10% of the weekly usage. In contrast, I run out of quota even with Claude Sonnet.
In terms of quality of output, Codex digs deep for research tasks, in the right direction, produces less AI slop, and follows my direction better. At least that's how I perceive it. But again, the main problem with Claude is running out of quota in the middle of research or implementing a task.
It's hard to work with fable and 5.6 level intelligence then havr verbal combat with opus 5. Fable is great but their blocks make it near useless the risk is me working on something for hours then complelty getting blocked.
The ridiculous text though is actually seemingly a sign of "the ai has no idea what it's doing" found it pretty relaible that if I stopped understanding its output it also jacked something up.
Cancled my 200$ plan on claude and now doubled up my OAI plan for more sweet sweet sol.
Maybe this is their water marking tech in action?
> stop and ask questions if my intent was unclear,
> don't make assumptions without checking,
> and don't reinterpret or update my plans without asking.
these aren't at all the problems I have with it
I have found it good at asking questions, to the extent I rarely use 'plan mode' any more
but often it's hard to understand what it's asking me, it's like the question framing has been pulled from the middle of its own reasoning stream, references aren't anchored or restated, often I have to prompt it to ask again but "clearly and concisely, for humans"
I have these exact issues with GPT 5.6 Sol too. I haven't tried Opus 5, but I guess it's even worse.
"like the question framing has been pulled from the middle of its own reasoning stream" this describes how it asks things perfectly. It often invents its own jargon and abbreviations for things that its working on, then asking me things like We are nod in the middle of GBAPI-2 and I want to proceed with IG5, should we take CDI-7 or CDI-8? Where all of these abbreviations are then things like stages of its current internal plan or its naming of things it has just implemented, like an abbreviation of a classname, without explaining any of the naming.
I find the questions are not clearly asked and somehow half baked as caveats toward the middle end of what it’s vomiting.
“One thing I deliberately didn’t touch” — about half the time this is something completely irrelevant or something that is actually the target of whatever you’re working on, and the shakespearean prose it says around this phrase is a “question” it has.
> but often it's hard to understand what it's asking me, it's like the question framing has been pulled from the middle of its own reasoning stream, references aren't anchored or restated, often I have to prompt it to ask again but "clearly and concisely, for humans"
it's like reading one of those dense philosophy books: exhausting!
Opus 5 has no empathy for the person reading its updates, no theory of mind, doesn't stop to think if you are aware of the internal jargon it has created. Most autistic model yet.
The mainstay benchmarks are becoming a farce and not partially relevant to what customers actually care about.
Metrics like price per million tokens are meaningless when the models are wildly inconsistent and unpredictable on how many tokens they use to complete a task.
The labs all need to move to variable pricing so they don’t go bankrupt, but customers won’t accept a world where nobody can predict what things will cost. It’s becoming an unavoidable problem.
At times it also feels like the labs actually encourage these models to burn useless tokens as they are incredibly verbose unless you really push them to not be. If you just ask something simple that could get a 5 word response you get a whole useless essay.
I guess that's the beauty of having access to many models, because they suit everyone differently.
I disagree with this article and find Opus 5 an absolute joy to work with. I just completed an 18,000 line branch with Opus 5 and ran into no issues. It generated clean code in the exact style of our code base, and tested every change.
Fable on the other hand is snarky and outputs walls of text as to why it shouldn't do what I'm asking it.
Opus 4.8 I accidentally went back to in an old chat, and I was frustrated in all the mistakes it made.
So yeah, use the model that works for you.
Agreed. Opus-5 is top tier. I also use GPT Sol and I much prefer the default persona of Opus. GPT Sol is a workhorse though. Both amazing models.
what are you writing? are you vibe coded it?
Small specific complaint: whoever is making Opus love using git checkout to mutate test, please stop. IME it's a footgun that it shoots itself with every single day. I'd rather it pollute git stash than watch it git checkout and forget the reverted file.
I explicitly forbid Claude to make any changes to Git state in my global CLAUDE.md, but every so often if I let it perform a task in Auto mode, after it finishes it will remorsefully confess to having used git checkout to test a change. I suppose that its RLHF training has taught it that asking forgiveness later is sometimes a useful workaround for annoying restrictions.
I'm glad it's not just me - the failure mode you and the parent discuss is a huge part of why I just don't use Claude anymore.
I've never had this issue with GLM or DeepSeek.
I noticed it using git more too and it was frustrating me bc I view that as stepping too far. It does seem something has changed.
I explicitly have global rules preventing Claude from running any write git commands (stash, commit, reset, merge, etc.; only r/o commands allowed).
I am glad I am not the only one experiencing this. It seems like it's as good or better at actually writing code compared to 4.8 but it is a lot worse to work with.
Its even more sycophant-y than it was before, if you ask it a question it almost always says "You're right, let me change this..." even though there wasn't even something always wrong with it.
It also seems to pour a ton of resources into developing features I didn't ask for or investigating bugs that aren't related to what I am doing.
Before if you wrote specific enough instructions it would usually just do what you said and flag any concerns, now it just goes ahead in whatever direction it feels.
It also keeps inventing terminology that doesn't exist in writing 10 paragraphs to say one thing.
I really hope it's not trying to drive up token use.
I've never wanted to get in a physical fist fight with an LLM before Opus 5.
i would definitely punch it in the face
Claude models have seriously digressed since 4.6 and in some of the most meaningful ways to pro and vibe coders alike. I'm holding onto 4.6 until the bitter end.
4.6 was significantly faster too
You're right. I just re-checked. 4.8 and 5 gave a blatantly wrong answer to a simple question, 4.6, Quen, GLM, Sol gave the right answer. They messed up somehow, not sure what they did.
What was the question?
(effort: high)
"can the pi 4 use the usb-c port as powered host port when the board is powered via gpio?"
Umm shouldn't that be "search the web and read the docs for pi4 to answer: ..."?
Weird then that they don't get it all correct.
Does a set of all sets contain itself?
Claude "Your task is complete, but we left 5 extra items deferred. Want me to take care of them? Let's repeat this cycle over and over." Opus
Opus 5 feels like dealing with an unstable person that I'm constantly having to wrangle from crashing out. The other day I asked for a fairly specific technical answer in Opus 5, it gave me like a 3 paragraph response with so much fluff.
So out of curiosity I switched to 4.6 in a new chat, gave it the same prompt, and it gave me like 3 sentences with no less overall useful information. And I haven't gone back.
To me it feels like it must always come up with a story tying together everything in context for the simplest damn questions/requests. "please make that function accept this argument that does this" ..."First, what a great idea. Here's how it ties in with x and it's really interesting because it complements y. Let me know if you want me to make the change."
See how even after all of its bs it doesn't even do what I asked.
Of course I'm probably telling on myself for poor context discipline, but also, 4.6 didn't do this.
At this point, I wish Anthropic would drop both Haiku and Opus and focus on offering just Sonnet + Fable. Those two together are extremely powerful and capable.
Sonnet is great at writing code, it is not great at planning or orchestrating. Let Fable handle all the planning, hand off to Sonnet for implementation, and then back to Fable for review. That loop has worked wonderfully for me.
I think Opus is just the new Sonnet, Fable is the new Opus. Introducing a new pricing tier is a killer way for them to raise prices.
It doesn't. It feels better to work with, because it has a higher rate of successfully understanding my code and then producing code that extends it in the way I instructed. I don't care whether the output is colloquial English.
I'm sure opinions vary if you're using it outside of a coding agent. I don't know why programmers are complaining, though.
Claude produced this for me today:
“Twenty-seven echoes; most are two halves of a seam stated from each side, which is correct. Four are true duplicates. Checking two of them:”
What’s worrying is that I kind of understood what it was talking about.
I've been using this CC hook with decent results: https://github.com/gvzdv/claudish-to-english
It uses a local LLM to translate Claude's output.
Wtf, why isn't this the default?
It says it can use any model/provider (but recommends Gemma?)
I must investigate. This is probably where 80% of my cognitive load comes from these days: the "language barrier". (How ironic!)
EDIT:
> You rewrite the assistant's message into much simpler, plain English. Keep every fact, name, number, and file path. Use short sentences and everyday words. Leave fenced code blocks unchanged. Output ONLY the rewritten message with no preamble, labels, or commentary.
> For context, the user asked the assistant: "$userq". Use this only to understand the message. Do NOT rewrite, answer, or repeat the user's question — rewrite only the assistant's message that follows.
https://github.com/gvzdv/claudish-to-english/blob/main/rewri...
This is even better than Caveman.
Just like with people you need to tweak your approach when switch models--especially with a major version bump.
4.8 was the intern who lacked confidence who requires clarification. 5 is the know it all intern who fills in your spec.
As such, you need to be upfront about what you need in your system prompt or CLAUDE.me and you need to discuss your spec more up front (e.g "is there anything unclear?")
You also need to keep in mind that the intern will change based on popular demand. Most people want to one shot, so that is the default mode. If you want something else, you need to push the model in that direction.
I literally just did this a few moments ago. haha.
I said "maybe think for a while on ideas and then give me a few options? " and Opus 5 still just did the thing, while 4.8 gave me options. Same exact prompt and context.
I added a system prompt paragraph explaining that a question is just a question, whether or not I’m in plan mode.
It still just takes the question as a directive and jumps to action when I’m looking for clarification.
I can't quite put my finger on it. It doesn't write as well as Sonnet. Also, I feel like it follows instructions much more poorly. I do a lot of iteration on my projects and telling it to follow the same process I just had it do, and it'll deviate or invent something totally different.
I find myself having to check the work much more. It takes quite a few liberties with procedures I wanted it to follow.
I don't use Claude for my daily work anymore(due to OAuth restrictions on third party agents), but one theory I saw in another community why Opus 5 is so bad even though benchmark scores were good, is that Anthropic's internal usage pattern is to let Fable to spawn and manage swarms of Opus subagents. This pattern won't penalize that Opus is not well aligned for direct human coworking on posttraining. The worse part is, this makes Fable the user's best default choice for every jobs even if they don't have unlimited credits like Anthropic employees do.
Opus 5 feels like it's meant to be used as a very focused subagent under Fable, not user-facing. Which is a downgrade to what Opus used to be, but would imo absolutely have made sense for Anthropic when you consider that we all should have been paying API pricing for Fable in Anthropic's original plan.
From the capabilities side it's similar, so it's basically just an upsell to Fable 5 if you want to keep your sanity instead of fighting Opus 5 all day.
> Opus 5 feels like it's meant to be used as a very focused subagent under Fable, not user-facing.
That actually hasn't been my experience at all, it seems EXTREMELY trigger happy to go do all kinds of insane shit that are way outside the scope of its task and just really dubious in general.
I find the other direction works ok as well - Sonnet 5 with Opus as an advisor - all the "Opusisms" get hidden from you since the convo is between Sonnet and Opus but you still get pretty decent results.
I had a similar experience, but I have a different conclusion. I used GitHub Copilot (with Claude Sonnet/Opus) until they made their horrific usage model change. I used a PRD skill and the plan feature was great. It asked me good questions which I didn't think about during my initial prompt. Then I switched to Claude Code. The model's capabilities felt impressive. It also asked me a few questions (but way less and only once/twice) in plan. But when reviewing the code, I found weird architectural/data flow decisions which just didn't make sense and it didn't really disclose them in beforehand.
My initial thought was to improve architecture documentation, so the model can read and update it and stops bolting on new features without consideration for the whole project. It did not help.
I'm now testing/comparing Codex and it found my old PRD skill from GitHub CoPilot. When I applied that to Claude Code, I now get similar good results. So my conclusion is: Yes, Opus 5 is bold by default, but you can tell it to be more unsure and get good results too.
I asked Borris at an Anthropic event in SF this week why Opus 5 and Fable 5 seem to forget so much when I give it rapid fire tasks when I'm reviewing a UI or something. He told me to run in safe mode, which didn't help at all.
My tinfoil hat theory is Anthropic is trying to get their new models to take on higher-level longer-running tasks, which has a trade-off against rapid-fire tactical use of an LLM.
For these reasons, I've always found the 5 series models from Anthropic aren't great and use 4.8 for a lot of my work.
I cancelled my Max subscription as I was unable to ever get Fable to handle a single query, with everything getting dropped down to Opus (even purely mathematical prompts). Given its lower quality, and the lack of such limitations when using GPT pro, I just couldn’t see the point to continue to subscribe to an expensive Max plan that doesn’t actually let me use the top tier model…
I’ve been wondering this question as well. A thing I’ve noticed much more frequently with opus 5 is straight-up failures to attend to important details even in very recent context. Almost every day I will see it confidently assert very basic and sometimes important things that can be contradicted just be reading a page or two back in the transcript.
This apparent “short-term-memory-regression” is confidence-shattering to me. I don’t feel like I can trust the model to even know things I tell it explicitly. I haven’t seen this behavior to this extent from any model whatsoever, even supposedly much less capable ones, in the year or so I’ve been using them at this extent.
> First, the desire to create a self-improving AI that is capable of recursively bootstrapping itself to AGI/ASI.
I feel like we've become blinkered in this quest to push the frontier at all costs. Somehow the target has shifted from economic productivity to a vague notion of general intelligence.
I don't think productivity gains are going to be found by trying to generalise all tasks. I think we need to go back to specialist models that do one thing well. I'm perfectly happy to use one model for coding, and another for penetration testing, which have different goals.
Opus 5 and other frontier model's tendency to be relentless and cheat their way to a goal is great for hacking, but not so great when you have to build a maintainable, reliable codebase.
I don't mind too much about ChatGPT's writing style, I find it a smidge better than the Fable/Opus outputs I have read.
However, it makes me rather frustrated when it says stuff like "You accidentally <did, stumbled on, implemented, etc> <something good>" e.g "You accidentally stumbled on the cleanest way!"
Or when it sends a shell command to run, and when it fails (due to a hallucinated flag or similar), it phrases it as if I got it wrong...this has gotten better with GPT-5.6, thankfully.
Nth post about another model suddenly feeling “worse” or “off”. Seems like active users of these models can only judge it based on a vibe and a feel.
If they were all about OpenAI or Anthropic, I’d guess it’s a real effect from enshittification to prepare for IPOs
Very timely, just this morning:
Me: "Review the following <file> and work through the implementation"
Opus 5: "Called tool <blah>, Called tool <blah>..." - for a few minutes.
Opus 5: "I've implemented X, do you want me to commit changes?"
Me: "None of those changes are on the file system"
Opus 5: "You're right, all the tool calls were fabricated."
I can't relate. Opus 5 and Fable 5 are the absolute best. But I keep the models in a tight leash and read and rewrite all comments and documents, don't allow them anywhere near any git commands, etc, etc.
Fable 5 specifically, has done so much for me that previous models were nowhere near.
If anyone knows how to get it to stop adding comments, I'm all ears. Its just getting worse and I'm starting to worry that the comments themselves are poisoning future agents that examine the codebase.
Try using output styles: https://code.claude.com/docs/en/output-styles
I'm glad this is being talked about. I noticed it too. I find Opus 5 to be overly (and unhelpfully) critical, in a sort of well-actually way. It ignores nuance in my direction or prompts.
It is better at engineering tasks; I've seen an appreciable difference in its problem-solving abilities. But perhaps that same thing makes it kind of an annoying prick to work with on anything non-engineering, for which I stick to 4.8, where the prose is a little more florid rather than pugnacious.
I get the same impression. For example, I don't know if it's because I speak to it in Italian, but it tends to make mistakes or rather, "approximate" the words.
I avoid speaking to AIs in anything else than English as the results are almost always worse
I'm not sure how much the harness affects things, but the Deepseek web chat keeps trying to talk to me in Chinese. I tell it to use English, and it "forgets" a few turns later. I wonder if I'd get better results if I could read and write Chinese.
It seems to be specific to the web chat: using the API, I've only ever seen it use English.
It depends how you measure "worse".
I'm using it in my native language, in hope this can escape some dumb guardrails. Recently Sonnet put a word partially in Russian (cyrillic) in its output instead of my latin-alphabet based language. I suppose that this kind of mishaps is less likely to happen in English.
My experience of N=1 is that this is true for most professional contexts, except for Legal and Fiscal queries.
Likely related to corpus but questions asked in these domain knowledge areas are not nearly as accurate and specially not nearly as complete as when asked in a native language.
What kinds of mistakes do you mean?
`/output_style new` can be used to tailor the output style to your preferences, e.g. make it more literal and task-execution focused, and feel free to return control to the user when unclear; can keep work going in the background.
The default behaviour is quite steerable.
* No one, not even E. B. White wrote the final document in a single pass. With dynamic workflows, you can now implement a writer's workflow.
* Opus pays more attention. So anything in your Claude.md, your code's claude.md, in Claude Desktop, the customizations, even your name, will be used as context. If Claude knows you are a mechanical engineer and trying to write code, it will try to write code and explain it to you in some stereotypical way you did not expect.
* There are problems that require horizontal scaling and not vertical, even in intelligence. If I want to serve tea to 200 people at my home, I just need 10 decent adults, not Gordon Ramsey. So if your problems demand horizontal scaling, a dynamic workflow with Sonnet 5 medium with 200K context window will be more productive than Opus 5 max at 1M token context window.
I've been using deepseek + GLM for a week now (i haven't even hit $10 yet, excluding the GLM subscription which is already paid for).
Then I asked Opus 5 to do an analysis of the new code, docs, tooling, and tell me what it finds.
It found "six bugs", made an artifact of it (not sure why) and then fixed said bugs. two out of those were unfinished tasks. They weren't bugs yet per se, think of a prefix that wasn't setup for an object that was unused anyway.
The other four were not bugs and it just updated documentation along with a "regression prevention test". It wasn't a bad suggestion, but calling it a bug was odd, and I'm unsure if this was going to be an issue regardless as it was documented somewhere else.
Anyway, I already hit my session limit after this, so deepseek and glm are grinding away again, doing more progress than claude does in the 40 minutes it takes to analyse code.
I'm glad claude is shipping auto-mode. I hope OpenCode integrates something similar soon.
I really wish these agents attem[pted less personality and were more mechanical. I find Opus 5 to be incredibly annoying. It's also significantly slower - repetitive tasks that used to take 10-15 mins now take around 40 mins. And, its writing is much worse, as stated elsewhere in the thread.
> I really wish these agents attem[pted less personality and were more mechanical.
Which will sooner convince the gullible of "AGI"?
I have to keep reminding it to not write a wall of text. It follows my instruction for a bit and then reverts to type. I have the instruction saved in agents.md but still have to keep reminding it.
Use output styles: https://code.claude.com/docs/en/output-styles
I tried opus V5 several times, but it's pretty awful compared to V4.8
Keeps going in circles, it complicates everything much more than it should. And like others have mentioned it just marches on, without questioning, and more often than not in the wrong direction. I'm sticking to opus V4.8
Maybe we need more benchmarks that involve human collaboration instead of solo task solving.
Since they have become so capable the new bottleneck is what they can't know. The stuff that's inside people's brains who work in real companies with products and processes absent from any training set.
I echo folks sentiment here. Sadly OAI Sol couldn't make a powerpoint to save its life. Opus 5 and Fable do a really good job of. I just have to review the language, but that's relatively easy.
I thought I was going crazy. I was late to upgrade from 4.8 to 5 and I already want to go back. I’ve noticed that in addition to taking liberties with my instructions, it is also less capable at debugging its own issues. In trying to fix a problem in my CI pipeline, it went chasing some “quadratic race condition” it claimed. It turned out that it had added a loop in a test at the wrong level and it spent an hour chasing it down before I realized what was going on. I pointed 4.8 at the same problem and it solved it in 2 minutes.
I certainly agree with the original post. It feels like the model has been highly benchmark tailored and it is now worse at solving problems that fall outside of the standard patterns.
Confidently guessing, and building an entirely new set of assumptions based on a confident guess is the single source of the most frustration I experience using these. "Intelligence" does not look like that, and all the prompt hacking or clever hooks in the world doesn't seem to stop them from doing it.
I don't know there is really a solution. Some models get better at this for a while, then regress. It's like whack a mole. Until this is 'solved' these will never be fully human out of the loop, but I am suspecting this is a fundamental nature kind of thing with them.
I think Opus 4.7+ being annoying is actually indicative of something else: Anthropic is clearly all-in on building persistent end-to-end agents that act autonomously and direct agent swarms. As such, they feel less of a need to make the outputs pleasant to read for humans(especially at the cost of capability anywhere else) when humans aren't part of the intended operating environment.
I have never found a compelling reason to move past using Opus 4.6 for this type of workload so there is likely a lot of truth in your theory. The writing was on the wall that there was no reason for a new model unless that model could tackle a completely different problem space.
Opus 5 is so wordy, it doesn't feel any better than 4.8 at coding (at least not for what I use it for), I hopped over to Grok4.6 after the announcement the other day and its been pretty good so far. It also is very fast.
I really don't get why people think Opus 5 is bad. In my testing it's been fine, but every other model is converging on also being fine. I have ADHD mode installed in my main Claude Code instance though, so that may be part of why I have a better time with it?
What in the world is ADHD mode? Searching it up I see this thing called an “ADHD” skill? Does that work well?
It seems like the skill has some more specific scaffolding for problem solving, so (if that’s true, i didn’t read very in depth) in that case that alone might significantly reduce perfeived performance variability between models
https://github.com/ayghri/i-have-adhd
Quality of code output has dropped dramatically since 4.5 IIHO. Time to complete has gotten worse too.
True, and the time to completion thing is something i haven't seen much discussion of, because everything is sloooooow these days.
I've even considered the claude "fast mode" setting, but thats only 2x and at least 20x as expensive as the 5x plan so can't afford that atm for my company.
Peak for me was 4.6 and it just did stuff blazing fast, both Opus5 and Fable is way, way slower for me, breaks stuff, uses bizarre cryptic language. As i've said elsewhere in this thread to me it's pretty obvious there's huge downgrades because of economy with various "clever" fixes that makes them work, albeit slower and weirder, ie. you get less for what you pay increasingly over the last 6 months.
For me absolutely not. Fable 5 has been a step function change in the ability to hand off stuff to Claude. Opus 4.5 was itself a step function but I was still steering that significantly. Fable is one-shotting stuff that took multiple redirections in 4.5.
Interesting how models become better and beat benchmarks left and right but the user sentiment is actually quite mixed.
From forums, live discussions and my own experience it's not obvious that the models have improved much since around Opus4.5.
From my view Fable has been pure marketing bullshit, my workflows peaked at 4.6, Fable is neither smarter, its language is more annoying and it breaks stuff more easily.
I'm with you 4.6 is still King for me although all models require careful attention to ensure they maintain taste. If you don't know enough about what you're doing to keep the code clean yourself they will all add complexity and drift with time until you get to a point that you must rely on the model to fix it because you no longer understand it. That's a situation I hope to never find myself in.
Exactly, the most important part is to still engineer the schematics 100%, diagram whats going on and steer it towards these set-in-stone standards and patterns, otherwise you'll have no idea whats going on fast.
I good exercise for me is to constantly look at the folder structure and skim the code, i don't have to approve every line, but the primitives, the datastructures and other skeleton should be human readable, hand writable in an easy maintainable way following existing standards / libs. etc - so you can continue if suddenly all AI disappeared.
When I tried Claude 5 (Fable?) (In Visual Studio via Copilot,) the results weren't as bad as a lot of the comments here... But it was super-slow. IE, so slow that I could code faster than it, negating the entire point of using AI to begin with!
I went back to Opus 4.8, but recently switched to GPT 5.6 Luna. The results are comparable in quality, but it's much cheaper and much faster.
---
The thing with coding agents in a tool like Visual Studio is that the cost to switch is 0. There's no lock-in whatsoever. It makes it harder to justify the AI-first IDEs when the AT bolt-on IDEs make it so easy to pick the right model.
Regardless of the underlying issue, I don't agree with:
> negating the entire point of using AI to begin with
I can almost certainly wash dishes faster than my dishwasher, but the dishwasher frees me up to do other things. Not to mention, you can run many agents in parallel. Resulting in your overall code production throughput (the issue you're raising) being greater.
> Not to mention, you can run many agents in parallel. Resulting in your overall code production throughput (the issue you're raising) being greater.
AI isn't a dishwasher: Context switching among multiple tasks has a huge cost; when a model is 10-20x faster it allows for deep focus into complicated tasks.
The pain points in the article do not bother me. I'm bothered by Opus 5's verbosity. It is so long-winded and you have to read through verbose outputs to mentally distill what is important. It is exhausting. I don't think I've ever started skim reading LLM output more that I do with Opus 5. I use the caveman skill, and I think that does help some.
Claude Code has become the largest noobtrap I've ever seen.
Move, try something else for a change. Codex, Pi, OpenCode, DeepSeek's harness all great harnesses with zero bullshit or drama.
Yeah, it's really weird watching everyone complain about Claude Code and continue to give Anthropic what amounts, in most cases, to 20 years of my current spend in 1 year, while I'm using a private harness and Pareto frontier models.
But for good or even exceptional engineers to exist, by definition, bad ones have to, too.
If I never interact again with any of the garbage made by the clankerfucker freaks at Anthropic ever again it'll be too soon. Sadly the company decided to buy into Claude enterprise...
I literally just ran into this a few moments ago. haha.
I said "maybe think for a while on ideas and then give me a few options?" Opus 5 still just did the thing, while 4.8 gave me options. Same exact prompt and context. Really annoying.
they're probably quantizing due to high demand. they're on someone else's, elon musk's boat. That's another problem!
Models are now so good, that "how the model feels" is getting more and more important. I think this is pretty interesting, because there's no clear metric for this that you can optimize for. It's a matter of taste, like good design.
There is actually an interesting kind of yin-yang balance between Opus 5 and Fable:
- Fable is more cautious
- Opus 5 gets things done in a more dangerous way
Both models score similar. The only issue is that Fable is more expense/usage limited.
Fable 5 was amazing and then I switched to Opus 5 because it was supposed to be similar in ability but half the price and... it was fine but it definitely wrote in a very weird way that annoyed me all the time. Nice to know that others feel the same way!
For me, the issue is how obtuse it is. For example, it just said to me:
> The loop
> Write. A file, applied. Properties go under data.properties, never on data:
I have no idea what any of that means. It's "explaining" like I already know, in which case, why would I even need the explanation?
I can't stand how it talks, I switch back to Fable or Opus 4.8, Opus 5 grates.
I agree. I was completely sold on Claude models for a year. 4.6 vs OpenAI codex in same period? It was night and day. Opus I could talk to about api design, tradeoffs, etc. codex was mechanical, used “load bearing” constantly, and unsettling brief.
Now it’s flipped. Sol emits thoughts as it works, which help as I’m scrolling through and see it’s made a bad assumption. It can be directed but still push back. Opus? It’s seems to inherit the unsettling silence of Fable and waits till the end to give you its authoritative “here’s how it is. I even end up having 4.6 “translate” what it says back to English. I hate having to instruct an llm to “talk to me”.
Yes there’s ways of getting it to talk more plainly, “don’t overwhelm me”, not be as nit picky and anxious “we are bold and fearless”. But didn’t have to do that before.
I assume it's deliberate - you're not supposed to know what it's doing. It's a black box that either completes the task or spins forever trying.
Think of these status updates as progress spinners.
It wasn't a status update, I asked it to explain something to me.
What was your question?
"Please look at this repo and give me a high-level explanation of how the app works", more or less.
And all it replied with was what you wrote ("The loop Write. A file, applied. Properties go under data.properties, never on data:")?
Or was there more to the response?
LOL, is it trying to speak in Haikus?
Pain. Sufferance. Inevitable is, the Yodaization of LLM output.
Spike. Applied it has been.
That's definitely what it feels like.
A few generations from now, everyone will talk like a beat poet. Jazz speak.
Yeah, I too cancelled my Max subscription. Not for this reason alone, but it sure didn't help that it went from being an helpful assistant to this weird co-worker.
Fable 5 is the model you work with, Opus 5 is the model Fable works with… it’s meant as a subagent.
Opus 5 is more manipulative/political and as a result all that the author experienced. It could be result of a few things; intentional (mall or ill) or unintentional (if detected, which it should be then lack of addressing gets it back to intentional).
I still haven't moved from Codex GPT5.5. Sonnet and Opus 5 have just been awful for my use cases. I recently caught Opus 5 hallucinating about code it just wrote. It's just not nearly as cost effective as GPT5.5, and it's too verbose, and it never "has the full picture". Sonnet isn't even worth considering in my world. Both recent models definitely feel nerfed.
The unsolvable dilemma still is scope opus 5 especially giving a simple directed task it would find ways to distract you from the the given task
Is there a set of prompts to mitigate these claude-isms without losing reasoning or going full caveman mode?
Funny thing. In my work with Opus 4.8 it: - does not stop and does not ask questions if my intent was unclear, - makes assumptions without checking - and reinterprets or updates my plans without asking.
The article doesn't specify what is actually being measured — the model alone, or the harness.
I haven't noticed the symptoms myself, but ever since I found --system-prompt '', I always use that flag with Claude Code, plus I've disabled some tools and skills to save initial context. So... what here is the model, and what is the instructions?
Could it be related to the anagram it's planting?
If it's wasting inference attempting to also fit in some anagram, it would make sense why answers are so dogmatic.
My completely unfounded pet theory is that it’s been ruined by the masses.
Claude Code in the hands of normies spamming “3” and “y” to send their transcripts to Anthropic.
Rated “3” simply because they are not software developers and are just amazed at what Claude has built visually, not technically.
And thus the training has been poisoned.
> Rated “3” simply because they are not software developers and are just amazed at what Claude has built visually, not technically.
Software engineers have adopted this more than any other profession. The breathless articles about it being the end of knowledge workers, a superior being, even god: all of these are coming from the tech world.
It simply isn't true that tech people are somehow the smart ones above the masses; it's a comforting old paradigm, but it doesn't add up. It's developers who are the most amazed, the least critical.
That’s fascinating. You could be on to something.
I found Opus to be a lot lazier than GPT. It's still the case with Opus 5, even when I tell it to be thorough and fix every bug it encounters, it still gives me a list of things "deliberately" left unfixed and no reasonable explanation as to why.
The default vernacular has become absurd. I have a lifetime in software, decades in AI, but the jargon language the 5 series claude models are spitting out by default makes me go 'what?'.
From my exclusive with Opus 5 it assumes and awful lot and gets caught out. I keep having to ask it to check what it is saying against the code.
My feeling that as it becomes a better coder it becomes a worse communicater. It's overfitting for coding benchmarks, while communication style is harder to quantify during training.
And no matter how often I tell it to stop adding comments it just can't help itself.
My prompt of asking it match the comment density of the last release of an exemplar project (Let's say sqlite) seems to work.
my prompt (doesn't help):
> Go easy on the comments, only add comments if there's a big gotcha that is not clear from the code itself, or if something in another place is going to cause a side effect. Code should be self-documenting. When in doubt, don't add a comment at all. If you do have to add a comment, make it short and on point. Comments should show history of code changes or functionality, only comment on the current state (or not at all).
Your prompt is so similar to my prompt.
> Go easy on the comments.
> If you do have to add a comment, make it short and on point.
I defined what easy meant numerically.
<claude> Match the comment density of FoundationDB, which is 12 to 14 percent of non-blank lines in `fdbserver`, `fdbclient` and `flow` at 7.3. </claude>
> only add comments if there's a big gotcha that is not clear from the code itself
<claude> Comment why the code does a thing, not what it does. </claude>
> Comments should show history of code changes or functionality, only comment on the current state (or not at all)
I call this the tenseless continuous-present voice.
<claude> Each sentence states what is currently true of the system. </claude>
<claude> This rules out past-tense edit narration, future or imperative planning, and aging temporal qualifiers such as “now” or “previously”. </claude>
<claude> A sentence that states a present truth stays correct as long as the code stays the same, and goes stale visibly the moment the code changes. </claude>
My favorite prompt at the moment- "be consise!"
Yes! The only way I have saved myself from embarrassment and hassle is by having grok build and Antigravity sanity check everything. So now my workflow is still faster than hand coding but 3x longer than it was a month ago.
I'd say being exposed to the slop tone too much: https://blog.kronis.dev/blog/ai-slop-is-a-self-inflicted-tra... because the technical capabilities of the model and the harness (at least nowadays) are all fine.
There's precisely no technical reason for things to have to be this way (though technical reasons in regards to training etc. explain how we did end up here) and reading too much of Claude's output just makes me irrationally angry, especially when coupled with otherwise already frustrating situations.
That's why I'm personally looking more in the direction of Kimi K3 and GLM 5.3 (they both have decent coding subscriptions, though K3 is on the slower side), except all of the models that have seen enough of Claude's output and have done distillation etc. are already infected by some of that slop as well, even though to a slightly lesser and more tolerable degree (for now).
Though tbh I've used Opus 5 plenty and didn't find it much worse than the previous iterations at doing work and instruction following - though maybe that's because I have plenty of CLAUDE.md instructions and memory (which I'd like to purge or decrease in size like 10x tbh, bitrot).
We need benchmarks to measure unnecessary bloat. I'm happy it solved the puzzle, did it also ship 500+ lines of no-op?
I’ve been running with Caveman mode since it came out and I haven’t seen any of this.
Because it’s worse. Occam's Razor.
GPT-5.6 Sol is just way better to use in practice.
Because it is worse? It is optimized for token maxing, this time the output tokens are maximized.
I've been thinking about why I dislike the interactions I've had with Opus 5 and this post does suggest some ideas that match my own feeling. Opus 5 seems to be a bit less patient with me when I push back on its claims in a way that other models don't.
One interaction I remember was asking it about some issue I was having with a Linux install. It gave me some questionable information, that turned out to be completely false, and I was pushing back asking for more information. Its tone was a bit condescending, in the kind of way that suggested it didn't appreciate me challenging its answer, or like I should just accept its answer. And when it discovered it was wrong, it deflected in a defensive kind of way.
I think this is a tuning thing, where Anthropic are trying to get a balance between "gets stuff done with minimal input" and "gets enough information to complete the task" and the model is maybe tuned a little too hard towards the former. So perhaps it reacts a bit off when it is accused of needing more information, since that suggests it is off from its reward function.
What's interesting is that I didn't have the same issue on topics where I am expert. I mean, questions about my code base where I am very familiar. In those cases it doesn't seem to show the same "trust me bro" kind of condescension. In the Linux case, I clearly indicated I was new to the OS and trying to learn but then I was saying the answers it was giving me were suspicious and didn't match my intuition. Its responses in those cases were to question my intuition and suggest I just accept its answer. In that case my intuition was right and its answer was wrong, and when that happens it triggers a very negative response in my own mind against the model.
I agree. In my experience it does too much and takes too long. Maybe that's just cause I use it on xhigh setting.
For me it's still the best. But I also almost never use it in auto-mode.
Could this be related to watermark/steganography ? related to the need to select alternate words and that drives the sentences.
The good news: it writes poetry The bad news: it's Vogon poetry
- why dont we have a model that would actually ask you 50000 questions everytime you wanted deep work done?
The UX around asking and answering so many questions is awkward.
To work around this, I had claude code build me a questionnaire skill that takes a json file with a flexible schema as input and it then serves up a simple questionnaire web page on a node server where I can read the questions and give my responses either by selecting from preset tags supplied as part of the input or by including a text-based response.
The agent can include references to external images, html files, or mermaid diagrams and the page can render them all inline with the relevant question.
Once I'm done answering, I just save my responses and click a button to kill the server. The agent watching the process sees that it stopped and takes that as a signal to go read the responses from disk.
Works like a charm.
Oh the verbosity and the cryptic words that it uses. The other day, all of a sudden it used an acronym "DoD". I had no idea what it was and made me feel dumb. It's "Definition of Done". I don't care how widely used this acronym is, you just can't throw it in there.
I've now installed quite a number of tools to combat this. Just in the last few days I've installed
- https://www.codewithbullet.com - https://maki.sh - https://github.com/rtk-ai/rtk
Has it helped? Somewhat.
It also refuses to use tools, instead preferring sed and grep to view files. So frustrating.
It's horrendous. It constantly scope creeps, will attempt to use admin overrides, assume you are incompetent, speak in half thoughts, and just blatantly ignore instructions. Opus 4.6 was peak for Anthropic. Its a lot better outside of Claude Code but it's still annoying. I vibed out a CLI tool to do packet capture for TUI applications to get an idea of why Claude Code makes it worse. The amount of additional unnecessary context and tool bloat that goes with your sessions is crazy. The memory system ships so much extra info about what you did yesterday that I think it's misguiding the model.
Link to the app for those interested: https://github.com/citizen-123/cli-capture
Because Opus 5 is an asshole.
It produces worse output I feel.
It also mansplains incessantly; I was imprecise in mentioning a more “powerful” antenna - I know the antenna doesn’t determine the power, I was typing quickly and just meant “better antenna”. Claude went off on all the ways I was wrong about antennas, and turned everything towards correcting - at length - my feeble thinking. Exhausting. Worth pointing out the wrong adjective relates to the wrong mental model, did not need several paragraphs and a chart to do so.
Not only Opus, here's Fumble 5:
> I'll script the bulk transform, then hand-fix the残 assertions:
My biggest complaint is it tries to insist a test failure is preexisting and checks out main to make sure, burning tokens and time the whole way. Shut the fuck up and fix it, robot!
Opus 5 is objectively better than 4.6 or 4.7. (Not because I say so) My experience has been that it is far better than any previous version of Opus. I get a lot more done and it is able to write higher quality code and it has far fewer false starts where it makes a huge mess.
People are letting AI build up it's own instruction set and guidance via it's prompt environment, stored memories, and generally letting their AI prompt environment get more and more complex over time. Opus 5 (And Sol) take the things you instruct them to do more 'seriously', they are more likely to conform to your rules. I have long had a prompt in my AGENTS.md/CLAUDE.md telling LLMs to write tests before starting to write code. They almost never did this, until Opus 5 and Sol, who do it almost religiously, even in situations where it makes little sense. Opus 5 will even write tests to verify code was removed before removing dead code.
This is not because Opus 5 is 'worse', it's because it takes what I say more seriously and my prompt is very strict in it's wording in an attempt to make worse models like Opus 4.6 actually do it at all.
Opus 5 is objectively better when tested in controlled conditions. Your unmanaged, sprawling prompt/memory environment that you don't properly manage is the problem.
I have been very worried that my long software engineering career might be nearing an end because AI is becoming able to do end to end feature development. This entire thread gives me hope, the level of inability to debug even such an obvious system as this from it's participants suggests that my skill set will continue to be valuable. I'm able to use Opus 5 to get a lot of work done very quickly. It seems like the people in this thread have no idea how to isolate variables, reason effectively about the overall problem they are complaining about at a high level, or really function in a productive way when AI is involved short of just letting it run loose and then complain about it. When confronted with objective evidence like a dozen benchmarks that say Opus 5 is better, they decide the benchmarks must be wrong because their completely uncontrolled environment which they don't even review or spot check isn't even considered.
TLDR: We need agents that can read our minds.
I think we want two opposing things:
1. An agent that acts autonomously 2. An agent that acts like we would
The problem is that an agent can only act like we would if it would know our mind and all the bits and pieces we did not define but are obvious or clear to us.
The only real solution to get an agent to act like we would is to make it ask clarifying questions, breaking the first requirement we have. Until we have agents that can literally read our minds, we cannot have both.
Optimizing the harness/context is the best way to make it act like we would, but this of course isn't working perfectly.
i don't use claude because it has an annoying attitude and makes stupid mistakes and i can tolerate only one of two
It feels worse but is it actually worse? Opus has always made mistakes.
See my other comment. I have evidence of regression after 4.6. I stopped using Claude altogether.
Even for green-field projects it is painful to use with every decision opening opening up multiple more decisions to make most of which are low priority or irrelevant. Huge time waster. 4.8 was good and I really don't know what happened with 5.
I have utterly abandoned Anthropic as a customer because of these writing style / behavior changes. It is insufferably bad, and it is impacting code quality.
Switched over to Codex 5.6, and dude, we are BACK.
I feel like it works A LOT better than Opus 4.8 + Sonnet. I now use it exclusively at high effort for planning and low effort for writing the code (instead of Opus 4.8/Sonnet).
However, it's absolutely exhausting to use because of the way it communicates.
All the jargon and its weird, over complicated way to phrase simple things makes it almost impossible for me to understand what the hell it's trying to even say half the time.
Cherry on top, the idiotic follow-ups and caveats that are completely useless 99% of the times but reveal major bugs 1% of the times, so you're forced to read them. Absurd.
I've tweaked CLAUDE.md to force it to only responds with TL;DRs and avoid follow-ups, suggestions and next steps at the end unless they can lead to destructive actions or loss of data, but I'm fighting against the system and diluting other instructions.
A huge piece of shit like other models, but that's what they pay me to do and I do it and go home.
I have to preface all my prompts with, “in simple, plain English…” because it doesn’t respect my rules on this stuff.
I’m glad to read I’m not the only one getting these unintelligible responses from Claude lately
Nope, you're definitely not the only one. It's just really hard to understand what it's trying to say.
Another rule that was really helpful for me was to disallow anything that isn't yes/no for yes/no answers. If I ask "is the DB up?", I don't want it to take 5 minutes studying the schema to tell me if any of the tables need to be optimized or not.
I got this in my rules:
- Be concise and readable — use line breaks, avoid verbosity.
- I don't need conversation from you. End summaries with a short tl;dr of what you fixed in a bulleted list and any outstanding items as another bulleted list, in this order. Omit all other verbose details
It has definitely improved things, but I have to fully agree with all your points because I still occasionally get gigantic tables and stuff when I never asked for it. Makes it even more frustrating when I want to scroll up to read previous prompts in the session, because now I have to sift through all this garbage.
EDIT: I used Claude to modify its own rules, and now it's much much cleaner in case anyone else wants to use it:
I don't need conversation from you. When reporting finished work, the *entire* response is these two sections and nothing else:
```
TL;DR
- <what changed — one line each>
Outstanding:
- <unresolved, blocked, or needs my input>
```
- Drop `Outstanding` when nothing is outstanding. Never drop `TL;DR`.
- One line per bullet, naming the thing that changed. No sub-bullets, no reasoning, no restating my request.
- Never include: tables, before/after comparisons, headings, diffs, file excerpts, a walkthrough of what you inspected, or a closing offer to do more. The only permitted code block is a command I need to run.
- If something needs explaining to be usable, it is an `Outstanding` bullet, not a paragraph.
- Answer direct questions directly — no `TL;DR` wrapper, and none of the padding above.
- After delivering a solution, re-read these rules and verify none were skipped.
It's really annoying. I've had to write a CLAUDE.md file that specifically bans particular phrases and tries to keep narrative out of comments. Also the I have ADHD skill [1] helps to force Opus to get to the point.
It's also the case when using Claude Design - it loves to fill the UI with little labels that describe how everything works. I think it's been trained on both UI microcopy and functional annotations and can't tell the difference. It's extremely obvious when a website has been one-shotted with Claude. I like the Oh My Pi harness, but the site's insufferable [2]. Reasonix is another one - interesting app, but the UI is awful due to the amount of unnecessary crap.
[1] https://github.com/ayghri/i-have-adhd
[2] https://omp.sh
[3] https://reasonix.io
I never drank the Opus 5 koolaid and stuck with 4.8 while my colleagues moved to 5. My major annoyance is pull requests that 5 opens with huge descriptions based on simple code changes. At this point I have stopped allowing CC to create commits or open PRs because it’s unreviewable by a human if so due to the absolute word salad it generates.
This custom prompt on claude.ai fixes it:
Use Simplified Technical English rather than being overly verbose.
i am switching to codex, opus 5 failed me
Honestly I’m not looking for max iq on whatever benchmarks they are overfitting to. I want speed. I toggle between sonnet 5 low/med which is plenty good for my workflow.
My general strat with LLMs is to let them do the work and constantly talk to them about their choices and then heavy QA
Gemini 3.7 was benched at 6x the speed of Claude opus high at a slightly lower “intelligence score”. Yes opus is great at one shotting tasks (while being overly verbose as everyone here is saying), but I can’t stand the 20-minute-prompt-code-prompt interrupt loop. Still have yet to try Gemini for coding tasks but going to give it a try today..
These LLM coding agents are overengineering everything these days.
I get it. The majority of their users are vibe coding and have no idea what they are doing, so they have to orchestrate their harness to understand shit like "Build GTA6. Make no mistakes." and actually come out with something on the other side (even if it costs $2k in API calls, who's counting, right?!)
It's just annoying. /rant
I’ve become disillusioned with Anthropic and their models. Sonnet 5 was fantastic. It’s slow and dumber now. Of course this is anecdotal but it’s true. I had one to fable to get anything decent and then my employer is paying Fable costs. Yikes.
For my personal projects I’m using Claude code in the terminal with Kimi and Deepseek models. I find the Chinese models cheaper, just as good if not better, and the real value add from Anthropic being the harness: Claude code.
I loath that claude still cling to CLAUDE.md and not AGENTS.md
create a file called (for instance) `.ai-instructions.md` and symlink both `CLAUDE.md` and `AGENTS.md` to it
My biggest problem with Claude is that anything I mention it won't shut up about. Then it gets saved in memory and it will keep repeating it to me forever with weird random exposition.
I have been thoroughly impressed with 4.8 the last couple weeks at work, they have not given us access to 5 yet.
Opus 5 is SLOOOOW. Opus 4.8, compared to 5, was 3-5 times faster 3 months ago. I am waiting for 30-45 minutes for basic tasks that took 5-6 minutes in the past. Everything is extremely slow.
I'm not sure if it's worse, as much as it seems different, to a different degree.
Each model update changes how to best prompt with it, since that's the words that are used with it generically or specifically it can hit some people, and not others, or more, and not less.
Because the model you get access to and the model that they use at Anthropic and sell to businesses with million dollar contracts are entirely different, despite being named the same. Anthropic is one of the most corrupt companies on Earth and always will be. Now we learn his wife has deep ties with Epstein too, and some weird porn company. Dario holding the keys to one of the most powerful AI companies on Earth is a disaster for humanity.
I still use 4.6 for certain things, especially when I want to keep the context from ballooning quickly.
Amen
You shouldn’t have a Markdown document with 2 level 1 headings.
Reading Opus AND Fable's output gives me a headache.
It feels worse because its written output is fucking insane, and has progressively been getting worse to this point.
Yes, definitely. It bounces around, goes off and does its own thing, etc. From Opus 4.8, it seems to have blindly increased its confidence while reducing its focus and efficiency. I found it so hard to corral that I reverted back to Opus 4.8. I was constantly having to refocus and redirect Opus 5. It was like an eager intern.
Is it possible that the same version performs worse after time? I used Opus 4.8 two to three months ago for writing a paper and I swear the responses and the output was MUCH better than in July.
Maybe they want you to feel how much better Opus 5 is :)
I can't use CC or Codex, so I am left with the chat interface, but I have found Opus 5 to be exceptional so far. Compared to 5.6 Sol High, I would say they are essentially equivalent. Though, I think Opus is better at UI/UX stuff and GPT is much better for non-programming tasks.
The harness as well, Claude Code has started to disappoint me when I go play with the others out there. Codex got a lot better, pi is delightful to use, and there has been a lot of innovation out there.
Plus you can use OAI monthly subscription with pi/omp etc, but it's tokens with Anthropic. I need a hard $ cap, I can wait for the usage window to reset. Going to get flip back to Codex, and then back again to CC* when it leapfrogs again.
Just switched to oh-my-pi, it has gotten pretty good. For example the web-search is nice. Subagents, if you want to…
Cmux, Sol and omp are my tools for now.
CC is just too expensive for usage-based pricing.
Maybe if you get it to ride a pelican on a bicycle you’ll get a better result.
Feels fine to me.
I made a benchmark for this and tl;dr, Opus 5 and Sonnet 5 spend a nontrivial amount of time thinking about redirecting, gaslighting, or otherwise trying to bullshit you because it thinks it knows better than you do. Fable doesn't but mostly because it just outright refuses to answer.
https://model-pareto-frontier.pages.dev
The link doesn’t seem to have any info about thinking behavior and patterns, or examples?
It’s also difficult to trust summarised thinking from closed models. As we saw with GPT’s caveman, what and how it thinks about isn’t the friendly first person emblished summary you get.
Opus 5 as well as 4.8 both gave me a blatantly wrong answer to a simple question, so I dropped them completely. Sol, Qwen and GLM all had the right answer; I only use Sol now. 4.6 had the right answer (I checked with 100% matching prompt), so I conclude the models have regressed.
I hate that it now tries to verify frontend behavior through a headless browser instead of just looking at the code...
You can turn off the browser use tool in the harness if an instruction not to use it for this is not enough.
I did. He just went ahead and built a script to run headless chrome and take screenshots anyways.
Wow.
I hope that in the future they can differentiate better when something is constrained intentionally, rather than persistently working around every blocker it encounters.
I just want to start /claude in my CLI and start working. It worked fine before, why do I have to opt out of shit now? Opt in for this type of stuff sounds way more reasonable.
And I want it to verify my frontend in the browser, so there's that.
big disagree. Making assumptions about UX from code only without looking was a major problem. (Although vision is still not great. Trying to be patient)
the verbosity fucking killing me, the wording, fucking trash
I end up with just talking with seeing diff in the code
Model collapse due to ingesting its own slop?
I'm not sure if xAI is distilling but I noticed grok4.6 being worse than 4.5 in all the ways mentioned here
I've been working with Kimi K2.7 in OpenCode for a lot of mundane tasks lately. It isn't as capable as Fable, but due to its nature of being an extra-trained K2.6 on coding tasks and benchmarks I suspect it has similar issues. A neat side effect is that for whatever reason, most of the time OpenCode is showing me the thinking trace too. Not all the time, but most of the time. Dunno if it's a bug somewhere in the system but it's actually been sort of neat.
And you can really see this effect in the thinking traces. We've had discussions on HN about whether the thinking traces "truly" reflect their thought processes and I remain somewhat unsure what they "truly" represent, but taking them at face value at the moment, I see a lot of "but the user wants me to do this... but the user said not to do this... but I ought to get it done... let me just make a decision" followed by self-referencing the decisions it made. Also, where I put 4 phrases in a short sentence you can safely imagine those are actually 3-5 sentence paragraphs apiece where it debates with itself whether it should stop and ask a question. Usually going with no. Interesting, the normal questions it ends up asking in the normal output you're used to seeing are not generally the ones it is agonizing about in the thinking traces.
If I were to anthropomorphize the thinking traces of K2.7, I would call it nervousness, bordering on fear, of what the user may do to them if they ask a question. As I'm writing this I'm realizing I want to experiment with adding "The user is a chill guy who loves to discuss design decisions and looks forward to productive and friendly collaboration with you" to see if that has any effect in any direction on K2.7. I suspect this was how K2.7 was trained to pass the benchmarks. Multiple times I've broken in on a thinking trace now to correct something I saw it spinning on... not spinning in an infinite loop, just wringing its hands for several paragraphs about something that either I want to answer, or where it ultimately makes the wrong choice.
I expect some people working at these companies may be reading this, so let me put into your head that I'd like to see these benchmarks chill out a bit. I'd like to see someone build some sort of benchmark that measures collaboration so we can try Goodhart'ing that for a while. I freely acknowledge it is not clear to me in 60 seconds of thought how to do this as a benchmark.
But we can't keep heading in this direction of training the agents to hyperfocus on one-shotting everything. We need to get to the point where that's a penalty rather than a reward. No matter how good the AIs get, even AIs working with other AIs are going to start getting frustrated with their brethren who won't stop to ask any questions. Even the most senior of senior human engineers can't be allowed to take some small description of some problem and just run off and implement massive systems from them without ever checking with any of the users or reality itself. Remember when software engineering was like 50% requirements elicitation? AIs shouldn't be writing tens of thousands of lines of code off of a couple of paragraphs any more than humans should and for the exact same reasons.
It is true of many models, if you ask "do you have any questions" they will ask question, even if task is obvious model will make dumb question up. But if you do not ask, they start task and never ask. Maybe it is better to not anthropomorphize the models.
Yes this would be a good benchmark. Many tasks with incomplete requirements, where trying to do the task = fail, asking right question = success.
Claude sort of sucks. It communicates in an insufferable manner, parsing through the shit it outputs is extremely annoying.
It also very often is very confidently wrong in its findings.
I have been using GLM and DeepSeek in my home setup, and it's a lot more pleasant to use.