Constitutional AI: the abstract describes training a harmless assistant with no human labels for harmful outputs, the only human oversight being a list of principles; a supervised phase of self-critique and revision; and an RL phase in which AI comparisons of pairs of answers train a preference model used as the reward.

The exact words it rests on

We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal

What the source said when we opened it, on 13 Oct 2026.

The source

Constitutional AI: Harmlessness from AI Feedback (arXiv 2212.08073)
arXiv · 2022-12-15

Open the sourcearchived copy

Checked

Checked by the notis newsroom on 13 Oct 2026, against the source above.

In the story

Dario Amodei wants outside testers inside frontier AI companies 14 Oct 2026