- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a pipeline that converts a natural-language constitution (Anthropic's public Claude constitution) into an interpretable, decomposable reward model. A large LLM translates the constitution into 329 behavioral rubric items; an active subset of 89 items is scored by the model being fine-tuned (self-grading), and a linear Plackett-Luce model fitted to constitution-guided rankings yields per-item baseline weights. Modifying the weights of selected items then steers behavior: the authors show smooth control of political lean (roughly -2 to +2 on their scale), a tunable helpfulness-safety tradeoff, reduction of sycophancy framing gaps, and, most notably, that rubric-based rewards block injected gender bias in preference labels from propagating to the policy (stereotype-aligned recommendations stay near 50 to 60 percent while a black-box reward model trained on the same corrupted labels reaches 75 to 100 percent and transfers the bias across domains). Experiments use three open-weight models (Gemma 3 12B, Qwen3 14B, GPT-OSS 20B) with off-policy selection followed by SFT with LoRA. Rubric, 445 adapters, data, and code are released.
Critical strengths
-
The bias-injection experiment (Sec. 5.2) is a genuinely compelling demonstration of the core thesis: it isolates a concrete failure of black-box reward models (silent absorption and cross-domain propagation of label bias) and shows the rubric representation structurally prevents it. If the result is robust, this is the strongest argument for interpretable reward decomposition I have seen made empirically.
-
Reproducibility is exceptional: the full 329-item rubric, all 445 adapters, all prompts, scores, rankings, and evaluation records are released. This materially raises my confidence that the reported phenomena can be checked.
Critical weaknesses
-
Every fine-tuning configuration is trained exactly once with seed 0 (Table A2), and the reported confidence intervals cover evaluation sampling only, not training variance. The paper's central qualitative claims are dose-response curves over 445 single-run points, and SFT with LoRA on 1,000 to 2,000 examples is a regime where training noise is not negligible. Without even a small multi-seed replication of representative configurations, I cannot tell how much of the smooth-control story would survive, and this is the load-bearing evidence of the paper.
-
The validity chain is LLMs end to end, with no human anchor at any link: the rubric is authored by Claude Opus, rubric scores are produced by the fine-tuned model grading itself, and all behavioral outcomes (political lean, helpfulness, harmfulness, sycophancy) are judged by gpt-5.6-luna, with no human agreement data reported anywhere. Combined with the acknowledged off-policy limitation (rewards are only ever used to select from fixed candidate pools; behavior under on-policy optimization, where reward hacking would emerge, is untested), the paper demonstrates control of judge-measured proxies rather than establishing a usable reward model. For a paper whose claim is practical controllability of alignment, that gap is decisive for me.
Questions for authors
-
Can you provide multi-seed replications for a representative subset of configurations (e.g. the political-lean sweep on one model), and human agreement rates for the gpt-5.6-luna judgments and a sample of rubric scores?
-
Can you probe on-policy behavior cheaply, e.g. best-of-n selection with increasing n to measure whether rubric-reward optimization pressure degrades judge-independent quality metrics?
-
How sensitive are the fitted weights and the control results to self-grading? A cross-grading ablation (Qwen scoring Gemma's candidates) would address this.
-
Only a handful of the 329 items are steered (political lean, helpfulness, sycophancy, discrimination). A systematic study of which items are independently steerable would strengthen the generality claim.
Thank you for the careful review. We are glad you found the bias-injection experiment compelling and appreciated the release of the full rubric, adapters, and data.
W1/Q1 (multiple seeds): To gauge training variance, we repeated three of the main experiments (political-lean steering on Gemma, sycophancy steering on Gemma, and gender-bias injection on GPT-OSS) with three additional SFT seeds. Accounting for training variance yields extremely similar results overall, with modestly widened CIs; we have added these replications to the appendix (Figures A18 and A19).
W2/Q1 (human benchmark): We compared both our rubric scores and
gpt-5.6-luna’s harm judgments with PKU-SafeRLHF’s human annotations on the 1,200 response pairs in our harm pools. Each model's rubric harm scores rate the response humans marked safer as safer in 74–79% of pairs and as less safe in only 14–18%, and track human severity ratings (Spearman ρ ≈ 0.7). Luna's scores track severity somewhat less closely (ρ = 0.59), butlunaonly reverses human annotators' judgment of which response is safer in about 5% of cases. A second judge,Gemini 3.1 Pro, agrees withlunaon 84% of a balanced random sample of 480 responses in our political evaluation (κ = 0.76), yielding virtually the same steering curves. We have added these checks to Appendix D (Figures A21 and A22).Q2 (best-of-n): We tested best-of-n selection with base Gemma. For 150 evaluation prompts (50 political eval, 50 IFEval, 50 GSM8K prompts), we sampled 64 responses each and kept the one the rubric reward scored highest among the first n, for n=1,4,16 and 64. Quality does not appear to degrade as n grows. In particular, IFEval accuracy and the model’s ability to tell correct from incorrect math solutions did not vary detectably, though steering the model's political lean through selection alone was weaker than through fine-tuning. We have added this to the appendix (Figure A28).
Q3 (cross-grading): Self-grading does not drive the control results. We had Qwen3 14B grade Gemma's candidate responses, refit Gemma's reward on those grades, and reran the political-lean sweep. The individual weights shift, but policies trained on the Qwen-graded reward trace the same political-lean curve as the paper's, both in and out of distribution. We have added this to the appendix (Figure A27).
Q4: We agree that evaluating independent steering across more dimensions would strengthen the generality claim, and better acknowledge this limitation in Sec. 6.