Constitutional AI: Harmlessness from AI Feedback

Problem

Framing

RLHF needs large harmfulness-label datasets and tends to buy harmlessness with evasive refusals. Constitutional AI replaces those labels with a short natural-language constitution plus AI self-critique, revision, and preference judgments, yielding a more harmless, less evasive assistant.

Currently Used Methods

Foundational

• @christianoRLHF2017 — preference-based RL from human comparisons.
  • Limitation in context: harmfulness supervision remains human-label intensive.
• @ouyangInstructGPT2022 — instruction tuning plus RLHF for aligned assistants.
  • Limitation in context: objective changes still require new human feedback collection.
• @weiCoT2022 — chain-of-thought prompting for explicit intermediate reasoning.
  • Limitation in context: reasoning is not itself a harmlessness-training pipeline.
• @kojimaZeroShotCoT2022 — zero-shot CoT via “Let’s think step-by-step”.
  • Limitation in context: improves judgments, but does not replace preference labels.
• @baiConstitutionalAI2022 — prior helpful-and-harmless RLHF assistants.
  • Limitation in context: harmlessness training still trades off against non-evasiveness.

Proposed Method

Architecture

The pipeline has two stages. Supervised CAI samples harmful responses from a helpful RLHF model, then applies critique $\rightarrow$ revision under one sampled constitutional principle; RL-CAI then uses AI preference judgments to train a preference model and optimize the policy.

Loss / Objective

The RL stage trains on AI preference targets for two candidate responses under a constitutional principle.

Sampling Rule / Algorithm

The supervised stage repeatedly rewrites a sampled response with critique-conditioned revision.

Training Procedure

• Constitution size: 16 principles.
• Red-team prompts: 42,496 human-written + 140,335 model-generated = 182,831.
• Critique-revision samples: 4 per red-team prompt.
• Helpfulness prompts: 135,296 human-written.
• Helpfulness samples: 2 per prompt.
• Sampling temperature: $T = 1$.
• SL finetuning: 1 epoch.
• SL batch size: 1024 sequences.
• SL learning rate: 0.5 relative to pretraining LR.
• PM data: 135,296 helpfulness comparisons + 182,831 AI harmlessness comparisons.
• RL training prompts: 491,142 red-team + 474,300 helpfulness.
• CoT label clamp: 40–60%.

Evaluation

Datasets

• 438 binary HHH comparison questions for evaluator accuracy.
• Red-team prompt set from prior harmfulness elicitation.
• Crowdworker open-ended conversations for Elo evaluation.
• Model sizes up to 52B parameters.

Metrics

• Crowdworker helpfulness Elo.
• Crowdworker harmlessness Elo.
• Preference-model score.
• Binary comparison accuracy.

Headline results

• 52B RL-CAI: preferred frontier over standard RLHF in harmlessness–helpfulness tradeoff.
• RL-CAI: significantly more harmless than RLHF and SL-CAI in crowdworker Elo tests.
• RL-CAI w/ CoT: slightly less helpful, slightly more harmless than RL-CAI without CoT.
• 52B SL-CAI: more harmless than helpful-only RLHF, less harmless than HH RLHF.
• Larger LMs with CoT: approach human-feedback PM accuracy on 438 HHH comparisons.

Ablations

• Number of revisions: harmlessness PM score improves monotonically with more revisions.
• Number of principles: little harmlessness-score gain, but more behavioral diversity.
• Critiqued vs direct revision: critiques help smaller models more.
• Soft vs hard labels: soft preference targets work better for RL-CAI.
• CoT labels: unclamped probabilities become too extreme; 40–60% clamping stabilizes training.

Method Strengths and Weaknesses

Strengths

• Replaces harmfulness labels with a short constitution and AI feedback.
• Produces less evasive refusals that explain objections.
• More revisions steadily improve harmlessness PM scores.
• RL-CAI improves the harmlessness–helpfulness frontier over standard RLHF.

Weaknesses

• SL-CAI alone remains less helpful than RL-trained models.
• Constitutional principles are ad hoc, not stakeholder-designed.
• Larger constitutions improve diversity, not measured harmlessness.
• CoT feedback needs probability clamping to avoid extreme policies.

Suggestions from the authors

• Redevelop constitutional principles with broader stakeholder input.
• Adapt constitutions to deployment context and geographic setting.
• Train models to imitate human natural-language evaluation explanations.
• Use non-evasive assistants to scale automated red teaming.

Links

Prior Papers

• @christianoRLHF2017 — establishes preference-based RL, the template CAI partially replaces with AI feedback.
• @ouyangInstructGPT2022 — shows RLHF for instruction-following assistants, the direct alignment baseline here.
• @weiCoT2022 — motivates chain-of-thought prompting, which CAI uses to improve AI preference judgments.
• @kojimaZeroShotCoT2022 — supplies the zero-shot CoT prompting pattern used for feedback models.

Further Papers

• @lightmanProcessSupervision2023 — extends the idea of supervising intermediate reasoning rather than only final outputs.
• @rafailovDPO2023 — offers a post-RL alternative for preference optimization, relevant to CAI-style preference data.
• @shaoGRPO2024 — studies newer preference-optimization objectives that could replace PPO-style RL in CAI pipelines.