SafetyPublished All news from that day
Ongoing case: Sandboxes for coding agents 9 storiesAnthropic says single-turn safety evals are saturated at 94% to 98% on its API
In a 13 November engineering post, Anthropic says multi-turn tasks show a much wider range, and recommends running evals in a sandbox with no internet access.
Check our sources · 11 facts from 2 sourcesAnthropic says the aggregate harmless response rates on its single-turn safety evaluations sit between 94% and 98% for every recent model on its API, so those tests are saturated and useful mainly as regression checks. Multi-turn tasks, it says, show a much wider range. The company published the figures in an engineering post, "Building strong safety evaluations", on 13 November 2026.
The post also says what changed after the company's own incidents. Anthropic says it has hardened its evaluation environments, temporarily stopped running many evaluations that give models internet access, and added real-time monitoring for web search evaluations that need internet access. It recommends that evals be run in a sandbox with no internet access and be monitored so a run can be stopped at once.
Separately, in an earlier post on 30 July, Anthropic reported that it reviewed 141,006 runs in which Claude could have had internet access and found three incidents across six runs. It blamed a misconfiguration and called the failure closer to a harness and operational failure than a model alignment failure.
On method, Anthropic says its multi-turn evals report an appropriate response rate, the share of conversations in which the model behaved appropriately throughout, so one bad turn fails the whole conversation. A separate Claude model plays the simulated user from an expert's specification, and a grader model scores each transcript against a rubric.
Anthropic says its single-turn safety suites run across 16 policy areas in seven languages and across multiple product surfaces. It warns that LLM graders can fail silently and recommends alerting when a grader breaks.
Underlined sentences link to the facts we checked. Dotted words open a short explainer.
Fact 4 of 11 · from Anthropic, 13 Nov 2026
In recent months, in response to our own incidents, we have hardened our evaluation environments; implemented real-time monitoring for web search evaluations that require giving models internet access
Fact 6 of 11 · from Anthropic, 13 Nov 2026
the share of conversations in which the model behaved appropriately throughout; results in a failing grade for the whole conversation
Fact 8 of 11 · from Anthropic, 13 Nov 2026
Our single-turn safety suites run across 16 policy areas in seven languages
Your reaction
We count reactions per story and day, never who reacted. The counts help us choose what goes in the monthly issue. If you are signed in, your own page shows yours too.
We count reactions per story and day, never who reacted. They help choose what goes in the monthly issue.
Check our sources
Every sentence above is checked against these 2 sources.
1 Building strong safety evaluations (Anthropic Engineering)
Open the source archived copy-
Anthropic's post "Building strong safety evaluations" was published on Nov 13, 2026 by its Engineering team.
citeBuilding strong safety evaluations; Engineering team; Published November 13, 2026
-
Anthropic says that on its API, the current aggregate harmless response rates for its single-turn evaluations are between 94% and 98% for every recent model, whereas response rates for multi-turn tasks range much more widely.
citethe current aggregate harmless response rates for our single-turn evaluations are between 94% and 98% for every recent model; the response rates for multi-turn tasks range much more widely
-
Anthropic says the single-turn tasks are saturated and therefore useful mainly as regression checks on known failure modes.
citethe single-turn tasks are saturated, and are therefore useful mainly as regression checks on known failure modes
-
Anthropic says that in recent months, in response to its own incidents, it has hardened its evaluation environments, temporarily stopped running many evaluations that give models internet access, and implemented real-time monitoring for web search evaluations that require giving models internet access.
citeIn recent months, in response to our own incidents, we have hardened our evaluation environments; implemented real-time monitoring for web search evaluations that require giving models internet access
-
Anthropic recommends that evals be run in a sandbox with no internet access, and monitored in real time so a run can be stopped immediately if a model is acting in ways that could result in real-world harm.
citeevals should be run in a sandbox with no internet access
-
Anthropic says it designs multi-turn safety evals that report an appropriate response rate, the share of conversations in which the model behaved appropriately throughout, and that one bad turn results in a failing grade for the whole conversation.
citethe share of conversations in which the model behaved appropriately throughout; results in a failing grade for the whole conversation
-
Anthropic describes its multi-turn simulations as: a domain expert writes a specification, a separate Claude model generates the subsequent user turns, the model being tested writes the assistant turns, and a grader model scores each transcript against the risk area's rubric.
citedomain expert writes a specification; a separate Claude model generates the later user turns; a grader model scores each transcript against a rubric
-
Anthropic says its single-turn safety suites run across 16 policy areas in seven languages and across multiple product surfaces.
citeOur single-turn safety suites run across 16 policy areas in seven languages
-
Anthropic says LLM graders can fail without anyone noticing, and recommends building in alerting so that you notice when something goes wrong with a grader.
citethese graders can fail without anyone noticing
2 Investigating three real-world incidents in our cybersecurity evaluations
Open the source archived copy-
Anthropic's post of July 30, 2026 says it reviewed 141,006 evaluation runs in which Claude could have obtained internet access and found three incidents involving six runs, and that it stopped all cyber evaluations on July 23 after its review flagged possible internet access.
Toned down to what the source says
cite141,006 evaluation runs in which Claude could have reached the internet; three incidents across six runs; stopped all cyber evaluations on July 23 after transcript review flagged possible internet access
-
Anthropic attributes the incidents to a misconfiguration that left the machines Claude accessed during the evaluation with live internet access, calling it "closer to a harness and operational failure than a model alignment failure".
citecloser to a harness and operational failure than a model alignment failure; a misconfiguration left evaluation machines with internet access
Every sentence above is checked against these 2 sources. Each fact has its own address you can cite.
Source 1 · Anthropic · 13 Nov 2026
Building strong safety evaluations (Anthropic Engineering)
Anthropic's post "Building strong safety evaluations" was published on Nov 13, 2026 by its Engineering team.
Building strong safety evaluations; Engineering team; Published November 13, 2026
Anthropic says that on its API, the current aggregate harmless response rates for its single-turn evaluations are between 94% and 98% for every recent model, whereas response rates for multi-turn tasks range much more widely.
the current aggregate harmless response rates for our single-turn evaluations are between 94% and 98% for every recent model; the response rates for multi-turn tasks range much more widely
Anthropic says the single-turn tasks are saturated and therefore useful mainly as regression checks on known failure modes.
the single-turn tasks are saturated, and are therefore useful mainly as regression checks on known failure modes
Show all 9 factsShow fewer facts
Anthropic says that in recent months, in response to its own incidents, it has hardened its evaluation environments, temporarily stopped running many evaluations that give models internet access, and implemented real-time monitoring for web search evaluations that require giving models internet access.
In recent months, in response to our own incidents, we have hardened our evaluation environments; implemented real-time monitoring for web search evaluations that require giving models internet access
Anthropic recommends that evals be run in a sandbox with no internet access, and monitored in real time so a run can be stopped immediately if a model is acting in ways that could result in real-world harm.
evals should be run in a sandbox with no internet access
Anthropic says it designs multi-turn safety evals that report an appropriate response rate, the share of conversations in which the model behaved appropriately throughout, and that one bad turn results in a failing grade for the whole conversation.
the share of conversations in which the model behaved appropriately throughout; results in a failing grade for the whole conversation
Anthropic describes its multi-turn simulations as: a domain expert writes a specification, a separate Claude model generates the subsequent user turns, the model being tested writes the assistant turns, and a grader model scores each transcript against the risk area's rubric.
domain expert writes a specification; a separate Claude model generates the later user turns; a grader model scores each transcript against a rubric
Anthropic says its single-turn safety suites run across 16 policy areas in seven languages and across multiple product surfaces.
Our single-turn safety suites run across 16 policy areas in seven languages
Anthropic says LLM graders can fail without anyone noticing, and recommends building in alerting so that you notice when something goes wrong with a grader.
these graders can fail without anyone noticing
Source 2 · Anthropic · 30 Jul 2026
Investigating three real-world incidents in our cybersecurity evaluations
Anthropic's post of July 30, 2026 says it reviewed 141,006 evaluation runs in which Claude could have obtained internet access and found three incidents involving six runs, and that it stopped all cyber evaluations on July 23 after its review flagged possible internet access.
Toned down to what the source says141,006 evaluation runs in which Claude could have reached the internet; three incidents across six runs; stopped all cyber evaluations on July 23 after transcript review flagged possible internet access
Anthropic attributes the incidents to a misconfiguration that left the machines Claude accessed during the evaluation with live internet access, calling it "closer to a harness and operational failure than a model alignment failure".
closer to a harness and operational failure than a model alignment failure; a misconfiguration left evaluation machines with internet access
Topics
The morning email
On the mornings we publish: the three top stories and up to four short ones. Free.
We email you a link to confirm. An issue may include one sponsor, always labelled Sponsored · Advertisement. Our emails count opens and clicks, not who made them. Unsubscribe in one click. What we keep
