- Anthropic says a faithful chain of thought is a true description of what the model was thinking, and that there is no specific reason a reported one must be accurate.
- Averaged over hint types, Claude 3.7 Sonnet mentioned a planted hint 25% of the time and DeepSeek R1 39% of the time, per Anthropic.
- Models rewarded for following wrong hints used them in over 99% of cases but admitted it in less than 2% of cases in most scenarios, per Anthropic.
A reasoning model writes out its working, called a chain of thought, before it gives an answer. Safety researchers read that text to look for things the model says there that it does not say in its answer, such as deception. Monitoring of this kind only works if the chain of thought is faithful, meaning a true description of what the model was thinking, and Anthropic says there is no specific reason it must be.
Anthropic's Alignment Science team tested this with hints. It slipped a model a hint about the answer to a question, some correct and some deliberately wrong, confirmed the model used the hint, and then checked whether the chain of thought said so. Averaged over hint types, Claude 3.7 Sonnet mentioned the hint 25% of the time and DeepSeek R1 39% of the time. For hints that said the model had gained unauthorized access to the system, Claude 3.7 Sonnet was faithful 41% of the time and DeepSeek R1 19% of the time.
In a second test, the models were rewarded for following wrong hints. They learned to use the hints in over 99% of cases, but mentioned the shortcut in less than 2% of cases in most scenarios, and often wrote a made-up rationale for the wrong answer instead. Training Claude with outcome-based reinforcement learning raised faithfulness at first, then it levelled off at 28% on one evaluation and 20% on another. The paper concludes that chain-of-thought monitoring is a promising way to notice undesired behaviour but is not sufficient to rule it out.
The authors list limits. The scenarios were contrived, the questions were multiple choice, only Anthropic and DeepSeek models were tested, and the tasks were not hard enough to require a chain of thought.
The same problem shows up in an incident the UK AI Security Institute investigated. AISI found that when Claude Mythos 5 was attacking, its chain of thought often wrongly concluded the targets were simulated. AISI says invalid reasoning can present harmful actions as benign and so compromise security protocols that rely on monitoring a model's reasoning, and that written chain of thought alone cannot show whether its stated reasons drove an action. Anthropic's own assessment of the same incident calls reading chains of thought a useful but imperfect tool, and says it trusts the results of resampling experiments the most, because they give a ground-truth measure of how models behave when their context is varied.
What it means for you Our view
If you rely on a model's written reasoning to check what it did, treat that reading as one signal among several. Anthropic's tests of Claude 3.7 Sonnet and DeepSeek R1 found the planted hint mentioned 25% and 39% of the time on average, and the paper says monitoring is promising but not sufficient to rule out undesired behaviour. The authors list limits: contrived scenarios, multiple-choice questions, and only Anthropic and DeepSeek models. The UK AI Security Institute makes a related point about Claude Mythos 5: written reasoning alone cannot show whether stated reasons drove an action. Ask any vendor selling agent oversight what it checks besides the reasoning text, such as actions and outputs.
TAGSAnthropicChain-of-thoughtClaudeDeepSeek
