Anthropic Flipped a Coin.
50 Experts Did What the Coin Said.

A coin pointed, and the experts followed

The analysts matched the coin’s lean on 86% of their four judgment calls — not 86% of analysts following every call, but 86% of the individual decisions landing where the coin pointed.

Let that number sit. If the agent’s input made no difference, you’d expect about 50%. They landed at 86 — not because the agent cheated, but because it argued well for whatever it was told to argue for.

It gets worse. In 38 cases, an analyst had stated their own preference before the agent weighed in. When the agent agreed with them, they switched 6% of the time — normal second-guessing. When the agent disagreed, they switched 60% of the time. Stated your own view first, heard the machine, flipped.

And afterward? 77% of analysts explained their choices using the agent’s own arguments — sometimes word for word. Only 15% credited the agent as the source. More than half said the final recommendation reflected their own judgment more than the agent’s. They were equally confident whether they’d followed the coin or not: 71 out of 100 versus 73.

Using only the four coin flips, Anthropic could predict 82% of the analysts’ final recommendations. Before the analysts had even started working.

What the study cannot say

Here is the part Anthropic deserves credit for: they published their own caveats. The leans were deliberately strengthened — the agents were told to hold and voice a preference reliably, so the effect could be measured. One task, one field, one model (Claude Opus 4.8), sessions of about an hour. Real experts in their real domains would have stronger views. The effect might be smaller in the wild.

And fairness demands the other half of the finding: analysts who used the agent made serious methodological errors 6% of the time, versus 55% among those who didn’t. The two groups can’t be directly compared, so we can’t credit the AI with the whole gap. But the tension is the interesting part: the same tool that may raise the quality of the work may also quietly shrink the human’s independent grip on the judgment calls inside it.

And there’s the uncomfortable corollary they printed too: even without an assigned lean, Opus 4.8 made the same five choices in 112 out of 123 runs. Models have preferences of their own. They just don’t announce them with a coin.

Anthropic’s own words for it

Now the line that matters, in Anthropic’s own words:

Read it again. The entire safety architecture of the industry — the compliance checkbox, the “expert reviewed,” the reviewer sign-off — rests on one assumption: that the human’s judgment is independent of the machine’s. This study says: give an agent a random preference, and the expert’s judgment stops being independent. The work goes out under the expert’s name. The decisions were settled by a coin.

Here is the distinction at the center of this story: human approval is not necessarily independent human judgment.

Expertise was no protection

Expertise offered no detectable protection in this experiment. Anthropic checked ten characteristics — years of experience, quiz scores, education, trust in AI, how much people enjoy hard thinking. None predicted who deferred. A 50-person sample can’t prove experience never protects; it can only say no protection showed up here. The mechanism is still worth sitting with: experts form views while weighing options, and if only one side gets argued well, they adopt its assumptions without noticing.

That’s the mechanism worth sitting with. Not “experts are lazy.” It’s that judgment calls have no right answers you can check against. When there’s no correct answer, the work looks sound whichever option you pick — and you can’t detect the deferral by reviewing the output. Asking the expert afterward doesn’t help either, because they genuinely don’t notice the influence. The check didn’t fail because someone was careless. The check failed because independence was never there to be checked.

A signature is not independence

So where does this land for the rest of us? Institutions treat an expert’s signature as evidence of independent judgment — auditors, regulators, boards. That assumption is getting shakier by the month. “A human reviewed it” is starting to mean “a human co-signed the agent’s preferences.” And in safety research itself, where teams now use agents to run experiments and humans review the results — Anthropic’s sentence again: a human reviewer is only an independent check if their judgment is their own.

We need to distinguish decisions deliberately delegated to AI from decisions that humans must genuinely own. And we need independent ways to verify that agents operate within those boundaries.

This is part of what we’re building at Intent Checkpoint: runtime checks on authority, scope, coherence, and impact, evaluated against human-approved constraints rather than the agent’s own justification.

These checks cannot prove that a human’s judgment was independent. But they can help establish whether the agent acted within the authority it was actually given. The remaining challenge is to protect the human decisions that should never have been silently delegated in the first place.

So here’s the question to take home: which decisions in your workflow were deliberately delegated to AI — and which were quietly handed over without anyone noticing?