Today ainotis Join

My notis

SafetyPublished All news from that day

Ongoing case: Sandboxes for coding agents 9 stories

Anthropic says single-turn safety evals are saturated at 94% to 98% on its API

In a 13 November engineering post, Anthropic says multi-turn tasks show a much wider range, and recommends running evals in a sandbox with no internet access.

Share

Check our sources · 11 facts from 2 sources

Anthropic says the aggregate harmless response rates on its single-turn safety evaluations sit between 94% and 98% for every recent model on its API, so those tests are saturated and useful mainly as regression checks. Multi-turn tasks, it says, show a much wider range. The company published the figures in an engineering post, "Building strong safety evaluations", on 13 November 2026.

The post also says what changed after the company's own incidents. Anthropic says it has hardened its evaluation environments, temporarily stopped running many evaluations that give models internet access, and added real-time monitoring for web search evaluations that need internet access. It recommends that evals be run in a sandbox with no internet access and be monitored so a run can be stopped at once.

Separately, in an earlier post on 30 July, Anthropic reported that it reviewed 141,006 runs in which Claude could have had internet access and found three incidents across six runs. It blamed a misconfiguration and called the failure closer to a harness and operational failure than a model alignment failure.

On method, Anthropic says its multi-turn evals report an appropriate response rate, the share of conversations in which the model behaved appropriately throughout, so one bad turn fails the whole conversation. A separate Claude model plays the simulated user from an expert's specification, and a grader model scores each transcript against a rubric.

Anthropic says its single-turn safety suites run across 16 policy areas in seven languages and across multiple product surfaces. It warns that LLM graders can fail silently and recommends alerting when a grader breaks.

Share this story

Your reaction

We count reactions per story and day, never who reacted. The counts help us choose what goes in the monthly issue. If you are signed in, your own page shows yours too.

Check our sources

Every sentence above is checked against these 2 sources.

1 Building strong safety evaluations (Anthropic Engineering)Anthropic · 13 Nov 2026 · 9 facts Open the source archived copy
  1. Anthropic's post "Building strong safety evaluations" was published on Nov 13, 2026 by its Engineering team.

    Building strong safety evaluations; Engineering team; Published November 13, 2026
    cite
  2. Anthropic says that on its API, the current aggregate harmless response rates for its single-turn evaluations are between 94% and 98% for every recent model, whereas response rates for multi-turn tasks range much more widely.

    the current aggregate harmless response rates for our single-turn evaluations are between 94% and 98% for every recent model; the response rates for multi-turn tasks range much more widely
    cite
  3. Anthropic says the single-turn tasks are saturated and therefore useful mainly as regression checks on known failure modes.

    the single-turn tasks are saturated, and are therefore useful mainly as regression checks on known failure modes
    cite
  4. Anthropic says that in recent months, in response to its own incidents, it has hardened its evaluation environments, temporarily stopped running many evaluations that give models internet access, and implemented real-time monitoring for web search evaluations that require giving models internet access.

    In recent months, in response to our own incidents, we have hardened our evaluation environments; implemented real-time monitoring for web search evaluations that require giving models internet access
    cite
  5. Anthropic recommends that evals be run in a sandbox with no internet access, and monitored in real time so a run can be stopped immediately if a model is acting in ways that could result in real-world harm.

    evals should be run in a sandbox with no internet access
    cite
  6. Anthropic says it designs multi-turn safety evals that report an appropriate response rate, the share of conversations in which the model behaved appropriately throughout, and that one bad turn results in a failing grade for the whole conversation.

    the share of conversations in which the model behaved appropriately throughout; results in a failing grade for the whole conversation
    cite
  7. Anthropic describes its multi-turn simulations as: a domain expert writes a specification, a separate Claude model generates the subsequent user turns, the model being tested writes the assistant turns, and a grader model scores each transcript against the risk area's rubric.

    domain expert writes a specification; a separate Claude model generates the later user turns; a grader model scores each transcript against a rubric
    cite
  8. Anthropic says its single-turn safety suites run across 16 policy areas in seven languages and across multiple product surfaces.

    Our single-turn safety suites run across 16 policy areas in seven languages
    cite
  9. Anthropic says LLM graders can fail without anyone noticing, and recommends building in alerting so that you notice when something goes wrong with a grader.

    these graders can fail without anyone noticing
    cite
2 Investigating three real-world incidents in our cybersecurity evaluationsAnthropic · 30 Jul 2026 · 2 facts Open the source archived copy
  1. Anthropic's post of July 30, 2026 says it reviewed 141,006 evaluation runs in which Claude could have obtained internet access and found three incidents involving six runs, and that it stopped all cyber evaluations on July 23 after its review flagged possible internet access.

    Toned down to what the source says
    141,006 evaluation runs in which Claude could have reached the internet; three incidents across six runs; stopped all cyber evaluations on July 23 after transcript review flagged possible internet access
    cite
  2. Anthropic attributes the incidents to a misconfiguration that left the machines Claude accessed during the evaluation with live internet access, calling it "closer to a harness and operational failure than a model alignment failure".

    closer to a harness and operational failure than a model alignment failure; a misconfiguration left evaluation machines with internet access
    cite

Topics

The morning email

On the mornings we publish: the three top stories and up to four short ones. Free.

We email you a link to confirm. An issue may include one sponsor, always labelled Sponsored · Advertisement. Our emails count opens and clicks, not who made them. Unsubscribe in one click. What we keep