How distillation works and when it becomes an attack

Anthropic's February 2026 post describes distillation as "training a less capable model on the outputs of a stronger one."

Feb 23, 2026 ... These labs used a technique called "distillation," which involves training a less capable model on the outputs of a stronger one.

Anthropic's post says "Distillation is a widely used and legitimate training method."

Distillation is a widely used and legitimate training method.

Anthropic's post says "frontier AI labs routinely distill their own models to create smaller, cheaper versions for their customers."

For example, frontier AI labs routinely distill their own models to create smaller, cheaper versions for their customers.

Anthropic's February 2026 post says "What distinguishes a distillation attack from normal usage is the pattern."

What distinguishes a distillation attack from normal usage is the pattern.

Anthropic's February 2026 post says "Illicitly distilled models lack necessary safeguards, creating significant national security risks."

Illicitly distilled models lack necessary safeguards, creating significant national security risks.

Anthropic's February 2026 post says three labs it names, DeepSeek, Moonshot and MiniMax, generated "over 16 million exchanges with Claude through approximately 24,000 fraudulent accounts."

We have identified industrial-scale campaigns by three AI laboratories—DeepSeek, Moonshot, and MiniMax—to illicitly extract Claude's capabilities to improve their own models. These labs generated over 16 million exchanges with Claude through approximately 24,000 fraudulent accounts

"Distilling the Knowledge in a Neural Network" is by Geoffrey Hinton, Oriol Vinyals and Jeff Dean and was submitted to arXiv on 9 March 2015.

[Submitted on 9 Mar 2015] ... Authors: Geoffrey Hinton, Oriol Vinyals, Jeff Dean

The paper's abstract says making predictions using a whole ensemble of models "is cumbersome and may be too computationally expensive to allow deployment to a large number of users".

Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets.

The abstract says Rich Caruana and his collaborators showed that the knowledge in an ensemble can be compressed into a single model "which is much easier to deploy", and that the authors "develop this approach further using a different compression technique."

Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique.

The abstract says "we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model."

we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model.

The paper says: "An obvious way to transfer the generalization ability of the cumbersome model to a small model is to use the class probabilities produced by the cumbersome model as 'soft targets' for training the small model."

An obvious way to transfer the generalization ability of the cumbersome model to a small model is to use the class probabilities produced by the cumbersome model as "soft targets" for training the small model.

The paper says "The relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize."

the trained model assigns probabilities to all of the incorrect answers and even when these probabilities are very small, some of them are much larger than others. The relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize.

The paper's example is that an image of a BMW "may only have a very small chance of being mistaken for a garbage truck, but that mistake is still many times more probable than mistaking it for a carrot."

An image of a BMW, for example, may only have a very small chance of being mistaken for a garbage truck, but that mistake is still many times more probable than mistaking it for a carrot.

The paper says: "When the soft targets have high entropy, they provide much more information per training case than hard targets".

When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases

The paper's softmax uses a temperature T, "normally set to 1", and "Using a higher value for T produces a softer probability distribution over classes."

where T is a temperature that is normally set to 1. Using a higher value for T produces a softer probability distribution over classes.

The paper says the same high temperature "is used when training the distilled model, but after it has been trained it uses a temperature of 1."

using a soft target distribution for each case in the transfer set that is produced by using the cumbersome model with a high temperature in its softmax. The same high temperature is used when training the distilled model, but after it has been trained it uses a temperature of 1.

Anthropic's 12 October 2026 post describes distillation attacks as "covert, industrial-scale campaigns to extract the capabilities of one model and replicate them in another."

Distillation attacks are covert, industrial-scale campaigns to extract the capabilities of one model and replicate them in another, without the authorization of the original model's developers.

The 12 October post says that in Anthropic's own test a pretrained Claude model trained on full reasoning transcripts from Claude Opus 4.8 scored 80.4% on SWE-bench Verified, against 71.8% when trained on summarised transcripts.

We tested how well a pretrained Claude model performed on the SWE-bench Verified coding benchmark after it was trained on transcripts from Claude Opus 4.8. When training on the full transcripts, the distilled model achieved 80.4%; with only the summarized transcripts, it achieved 71.8%.

The 12 October post says that, by Anthropic's own estimate on its own platform, after it introduced the classifier the number of reasoning tokens extracted by distillers is "consistently 10 to 100 times lower than it was before."

To measure the effectiveness of our system, we estimate how many tokens of Claude's reasoning distillers have extracted on our own platform. After introducing the classifier, the number of reasoning tokens extracted by distillers is consistently 10 to 100 times lower than it was before