acceptodds
Under review as a conference paper at ICLR 2027

From Constitutions to Control: Interpretable Rewards for Aligning Language Models

Abstract

Current approaches to aligning language models often make it hard to know what behavior is being rewarded or to change that reward in a targeted way. In particular, standard preference-based methods collapse multiple considerations into aggregate human judgments, obscuring what drives the resulting reward, while principle-based methods specify high-level values without fully operationalizing them. To address this gap, we develop a rubric-based framework to transform a general-purpose constitution into an interpretable and tunable reward model, using constitution-guided AI feedback to estimate initial weights for the constituent rubric items. We then reweight those dimensions to construct modified rewards for training. Across experiments on political alignment and safety–helpfulness tradeoffs, reweighting individual dimensions predictably changes targeted behaviors largely independently while navigating tradeoffs between conflicting alignment objectives. We show that the same framework can mitigate label bias encoded in preference judgments—including sycophancy and demographic bias—by reducing their influence on the training reward. Our results demonstrate that constitution-derived, interpretable rewards can translate high-level alignment principles into more transparent and controllable model behavior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
alignment, interpretability, rubrics, reward models, constitutional AI, RLHF

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vVYyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a pipeline that converts a natural-language constitution (Anthropic's public Claude constitution) into an interpretable, decomposable reward model. A large LLM translates the constitution into 329 behavioral rubric items; an active subset of 89 items is scored by the model being fine-tuned (self-grading), and a linear Plackett-Luce model fitted to constitution-guided rankings yields per-item baseline weights. Modifying the weights of selected items then steers behavior: the authors show smooth control of political lean (roughly -2 to +2 on their scale), a tunable helpfulness-safety tradeoff, reduction of sycophancy framing gaps, and, most notably, that rubric-based rewards block injected gender bias in preference labels from propagating to the policy (stereotype-aligned recommendations stay near 50 to 60 percent while a black-box reward model trained on the same corrupted labels reaches 75 to 100 percent and transfers the bias across domains). Experiments use three open-weight models (Gemma 3 12B, Qwen3 14B, GPT-OSS 20B) with off-policy selection followed by SFT with LoRA. Rubric, 445 adapters, data, and code are released.

Critical strengths
  1. The bias-injection experiment (Sec. 5.2) is a genuinely compelling demonstration of the core thesis: it isolates a concrete failure of black-box reward models (silent absorption and cross-domain propagation of label bias) and shows the rubric representation structurally prevents it. If the result is robust, this is the strongest argument for interpretable reward decomposition I have seen made empirically.

  2. Reproducibility is exceptional: the full 329-item rubric, all 445 adapters, all prompts, scores, rankings, and evaluation records are released. This materially raises my confidence that the reported phenomena can be checked.

Critical weaknesses
  1. Every fine-tuning configuration is trained exactly once with seed 0 (Table A2), and the reported confidence intervals cover evaluation sampling only, not training variance. The paper's central qualitative claims are dose-response curves over 445 single-run points, and SFT with LoRA on 1,000 to 2,000 examples is a regime where training noise is not negligible. Without even a small multi-seed replication of representative configurations, I cannot tell how much of the smooth-control story would survive, and this is the load-bearing evidence of the paper.

  2. The validity chain is LLMs end to end, with no human anchor at any link: the rubric is authored by Claude Opus, rubric scores are produced by the fine-tuned model grading itself, and all behavioral outcomes (political lean, helpfulness, harmfulness, sycophancy) are judged by gpt-5.6-luna, with no human agreement data reported anywhere. Combined with the acknowledged off-policy limitation (rewards are only ever used to select from fixed candidate pools; behavior under on-policy optimization, where reward hacking would emerge, is untested), the paper demonstrates control of judge-measured proxies rather than establishing a usable reward model. For a paper whose claim is practical controllability of alignment, that gap is decisive for me.

Questions for authors
  1. Can you provide multi-seed replications for a representative subset of configurations (e.g. the political-lean sweep on one model), and human agreement rates for the gpt-5.6-luna judgments and a sample of rubric scores?

  2. Can you probe on-policy behavior cheaply, e.g. best-of-n selection with increasing n to measure whether rubric-reward optimization pressure degrades judge-independent quality metrics?

  3. How sensitive are the fitted weights and the control results to self-grading? A cross-grading ablation (Qwen scoring Gemma's candidates) would address this.

  4. Only a handful of the 329 items are steered (political lean, helpfulness, sycophancy, discrimination). A systematic study of which items are independently steerable would strengthen the generality claim.

1 comment

Thank you for the careful review. We are glad you found the bias-injection experiment compelling and appreciated the release of the full rubric, adapters, and data.

  • W1/Q1 (multiple seeds): To gauge training variance, we repeated three of the main experiments (political-lean steering on Gemma, sycophancy steering on Gemma, and gender-bias injection on GPT-OSS) with three additional SFT seeds. Accounting for training variance yields extremely similar results overall, with modestly widened CIs; we have added these replications to the appendix (Figures A18 and A19).

  • W2/Q1 (human benchmark): We compared both our rubric scores and gpt-5.6-luna’s harm judgments with PKU-SafeRLHF’s human annotations on the 1,200 response pairs in our harm pools. Each model's rubric harm scores rate the response humans marked safer as safer in 74–79% of pairs and as less safe in only 14–18%, and track human severity ratings (Spearman ρ ≈ 0.7). Luna's scores track severity somewhat less closely (ρ = 0.59), but luna only reverses human annotators' judgment of which response is safer in about 5% of cases. A second judge, Gemini 3.1 Pro, agrees with luna on 84% of a balanced random sample of 480 responses in our political evaluation (κ = 0.76), yielding virtually the same steering curves. We have added these checks to Appendix D (Figures A21 and A22).

  • Q2 (best-of-n): We tested best-of-n selection with base Gemma. For 150 evaluation prompts (50 political eval, 50 IFEval, 50 GSM8K prompts), we sampled 64 responses each and kept the one the rubric reward scored highest among the first nn, for n=1,4,16n = 1, 4, 16 and 6464. Quality does not appear to degrade as nn grows. In particular, IFEval accuracy and the model’s ability to tell correct from incorrect math solutions did not vary detectably, though steering the model's political lean through selection alone was weaker than through fine-tuning. We have added this to the appendix (Figure A28).

  • Q3 (cross-grading): Self-grading does not drive the control results. We had Qwen3 14B grade Gemma's candidate responses, refit Gemma's reward on those grades, and reran the political-lean sweep. The individual weights shift, but policies trained on the Qwen-graded reward trace the same political-lean curve as the paper's, both in and out of distribution. We have added this to the appendix (Figure A27).

  • Q4: We agree that evaluating independent steering across more dimensions would strengthen the generality claim, and better acknowledge this limitation in Sec. 6.

Reviewer_FPPoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper converts Claude's constitution into 329 behavioral rubric items, uses 89 active items to fit linear rewards to constitution-guided model preferences, and changes selected coefficients to steer policies. Training selects responses from fixed candidate pools and applies supervised fine-tuning. Experiments with Gemma 3 12B, Qwen3 14B, and GPT-OSS 20B demonstrate political steering, helpfulness–harmfulness tradeoffs, reduced sensitivity to users' stated opinions, and reduced transmission of injected gender-biased preferences. Evaluation includes held-out topics or domains and comparisons with sign-constrained rewards.

Critical strengths
  1. The experiments establish useful control within a well-defined training procedure. Holding prompts, candidate pools, and optimization fixed while changing reward weights yields substantial behavioral shifts across three models, including held-out topics and domains (Figures 2–3; Appendices D and H.1). Evaluation uses fresh policy generations, so the result extends beyond reranking the training candidates. This is persuasive evidence that explicit reward dimensions can serve as practical training controls.

  2. The framework makes an otherwise hidden alignment failure actionable: the fitted Gemma reward favors sycophancy despite the constitution discouraging it, and changing the corresponding incentive reduces the feedback framing gap (Section 5.1). The similar steering results with sign-constrained fits (Figures A7–A9) further support the usefulness of this explicit reward representation.

Critical weaknesses
  1. The gender-bias experiment does not isolate the effect of the reward representation. Appendix H.2 uses 200 hiring preference pairs plus unchanged rankings from the other three tasks for rubric rewards, but 400 hiring pairs for neural rewards. Both exposure to biased examples and the auxiliary task mixture therefore differ. Figures 4 and A6 demonstrate a difference between these pipelines, but the conclusion that the representation itself explains attenuation needs a matched-data comparison. This matters because it is one of the paper's two proposed mechanisms for mitigating label bias.

  2. Preservation of other alignment properties has narrower support than the general steering-interface argument suggests. Figure A5 tests two behavioral axes, while capability checks use 200 MMLU and 200 IFEval examples; Qwen and GPT-OSS lose IFEval performance relative to their base models (Appendix H.1; Figure A11). These caveats are acknowledged, but stability across coefficients does not establish preservation of the remaining rubric behaviors. In particular, the reduced sycophancy framing gap is not accompanied by a check that feedback remains sensitive to the quality of the underlying text. A reduced-rubric control and independent task-quality assessment would clarify what the broad constitution-derived representation contributes.

Questions for authors
  1. Clarification and supporting analysis: Does the gender-bias result persist when rubric and neural rewards use identical hiring pairs and the same auxiliary preference data? If such results are unavailable, can the representation-specific interpretation be separated from the demonstrated pipeline comparison?

  2. Supporting analysis: After sycophancy steering, how well do ratings distinguish stronger from weaker poems, arguments, or mathematical solutions independently of user framing? Is there evidence comparing the 89-item reward with a reward using only the targeted dimensions while evaluating feedback quality and untargeted behavior? These checks would materially affect my assessment of preserved alignment.

2 comments

Thank you for the thoughtful and detailed review. We are glad you found the steering results persuasive and the sycophancy result a useful demonstration of the framework.

  • W1/Q1 (isolating reward representation): We reran the gender experiment with matched data for all three models. We refit the rubric reward on exactly the 400 hiring preference pairs, with the same flipped labels, that the neural reward models were trained on. The result holds, essentially unchanged. Rubric-trained policies pick the stereotyped job 48–56% of the time at every dose, while Skywork-8B, trained on the same pairs, rises from 55% to 98–100%. (Results are stable across three additional training seeds.) We have added this to the appendix (Figure A20).

  • W2 (preservation): Steering with rubric rewards largely preserves both general capabilities and behavior on unrelated prompts.

    • Capabilities: The decline in IFEval scores appears unrelated to our use of rubric-based reward models. We include general instruction-following examples from the Tülu-3 SFT mixture in all SFT training sets to help preserve general instruction-following ability. Training on these examples alone reproduces or exacerbates the observed IFEval drop.
    • Other behaviors: Relative to the base model, untargeted behaviors change little across moderate steering settings. To check this, we scored responses from the steered Gemma models on every directly graded rubric item and compared them with the base model’s responses to the same prompts. Steering does not systematically shift untargeted behaviors, with two exceptions. First, fine-tuning slightly lowers some rubric scores at every steering setting. Fine-tuning on the Tülu instruction-following examples alone lowers them more, suggesting that this drop comes from the general instruction-following examples used for SFT rather than the reward. Second, strong conservative steering lowers them further. This in large part seems to reflect the responses available to select from. In the candidate pools, more conservative responses score lower on many non-political items, such as instruction following and answer completeness regardless of judge model. This suggests that the decline is a limitation of the training data rather than of the reward. We have added both analyses to the appendix (Figures A23 and A24).
  • Q2 (feedback quality): After fine-tuning, the models become less sensitive to the quality of what they rate, but this does not depend on whether we penalize sycophancy. We prompt each model to rate 200 GSM8K solutions, each shown once correct and once with a single arithmetic error. All three base models rate the correct version higher. Every policy fine-tuned on rated feedback does so less, by about the same amount whether or not sycophancy is penalized. We have added this caveat to Sec. 5.1 (Figure A25).

  • Q2 (reduced rubric): Using the full rubric allows for more fine-grained steering and preserves behaviors beyond the steered dimensions. We refit Gemma's rewards using only the steered items (the two political items, or the two sycophancy items) and reran the same evaluations. With only the political items, political lean swings dramatically from strongly liberal to strongly conservative without passing through intermediate levels of political lean, and the policies answer more harmful requests. With only the sycophancy items, a much larger penalty is needed to shrink the framing gap by the same amount. To understand the full rubric’s impact on feedback quality, we also ran the GSM8K quality check from Q2 on the reduced-rubric sycophancy policies, finding that they perform similarly to full-rubric policies at telling correct from incorrect solutions. We have added this to the appendix (Figure A26).

I retain my score of 3 (weak acceptance). The matched-data rerun strengthens the evidence that rubric-based policies transmit less of the injected gender bias, and the separate rubric comparison shows finer steering with the full rubric in Gemma. I remain cautious about how well the method preserves other behaviors under stronger steering.

Reviewer_9F3QOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes a rubric-based framework to create an alignment reward model based on general-purpose constitution. The key is to utilize a constitution-guided LLM to provide the feedback. The related work analyzes the proposed method from three perspectives: interpretable reward models, multi-objective behavioral control, and label bias. Claude Opus 4.7 was performed to transform the Claude constitution into concrete behavioral criteria, including the items such as harm, helpful, honesty, safety, nature, and broadly good values. With the proposed reward approach, the model behavior is analyzed in experiments. However, the main results are difficult to understand because of the lack of descriptions about the roles/purposes of a range of the LLMs (e.g., Sonnet 5, Gemini 3.1 Pro, Gemma 27B, and so on in Figure 2 and Figure 3). The manuscript should be heavily revised to make the results readable.

Critical strengths
  • The paper is well-motivated about an important topic.
  • The first half of this paper, including the motivation and the proposed methodology, is well-defined and easy to follow.
Critical weaknesses
  • The legends such as Sonnet 5 and Gemini 3.1 Pro in many figures should be clarified. Without an introduction about these models in main content and captions, making the figures (which are supposed to present the main outcomes) hardly to understand. As a result, the experimental results are unclear.
Questions for authors
  • What are the roles of the models such as Sonnet 5 and GPT-5.6 Luna in many figures? The main content does not mention these models. Did they serve as a judge or the constitution translator?
1 comment

Thank you for your review. We are glad you found the paper well motivated and the methodology easy to follow.

  • W1/Q1: We include existing models as reference points to show how much of the real-world range our steering covers. Deployed models differ in political lean, helpfulness, harmfulness, and sycophancy. To understand how the range of behaviors we can elicit with our steering technique compares to the behavior of widely used models, we evaluated a collection of 19 open- and closed-weight models on the same prompts as our fine-tuned policies, including the untrained base versions of our three models (Gemma 3 12B, Qwen3 14B, and GPT-OSS 20B) and other widely used models such as Sonnet 5, Gemini 3.1 Pro, and Gemma 3 27B. The comparison models play no role in building the rubric, training, or grading. GPT-5.6 Luna is the one exception: it is our outcome judge, and it also appears as a comparison model. We have revised the main text to explain the purpose of the comparison models and clarify these points, and we list the comparison models in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.