acceptodds
Under review as a conference paper at ICLR 2027

Character Training with Dueling Constitutional Optimization

Abstract

Learning from preference feedback is a critical step in producing AI assistants aligned with human values. Reinforcement learning from human feedback (RLHF) and Direct Preference Optimization (DPO) have emerged as standard approaches, with Constitutional AI scaling supervision through AI feedback guided by natural language principles. However, these popular methods make preference optimization tractable by assuming preferences are generated from latent rewards through a known transfer function, such as the logistic link in the Bradley-Terry model, and fitting an explicit or implicit reward function amenable to gradient-based optimization. In this work, we show that in Constitutional AI, this intermediate reward modeling step, explicit or implicit, can be dispensed with entirely. We introduce Dueling Constitutional Optimization (DCO), which treats a feedback model as a comparison oracle queried online through duels, and optimizes the latent reward function induced by a human written constitution directly through comparison-based stochastic search. We apply DCO to character training, where the constitution specifies virtuous dispositions we wish the model to internalize. Our results demonstrate that DCO outperforms both inference-time activation steering and training-based alternatives with PPO-based RLHF and DPO for cultivating character traits, while being simpler to implement and more memory efficient.

Primary area:
foundation or frontier models, including LLMs
Keywords:
constitutional ai, large language models, preference learning, alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_y4fLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DCO as a technique to learn from model constitutions. DCO uses Noisy Comparison Random Search to first perturb model weights, rollout from the perturbed and original weights, and then compare the responses with a constitution-informed judge. If the judge accepts the rollouts from the perturbed weights more often than the original weights, the perturbation is kept. They show that this approach can beat DPO and PPO based approaches for constitutional AI. In addition, they also provide some theoretical analysis of batch decision reliability, and an explanation of why they use ordinal feedback instead of scalar score feedback.

Critical strengths
  1. The improvements over DPO and PPO are large.
  2. The algorithm is simple but effective in this application area.
  3. The presentation of the results is quite nice.
Critical weaknesses
  1. The main weakness to me is the level of novelty. This paper is an application of Noisy Comparison Random Search applied to constitutional AI style optimization (with some small tweaks like the truncation operation). To me this seems like the application of an already existing optimization approach to a well known problem, and I do not see it as a significant contribution to the community.
  2. The tasks and evaluation benchmarks are all author designed. As the methodology is not novel, the only contribution I could see are strong empirical results. The empirical results are strong, but they are on author designed benchmarks which makes it hard to evaluate their significance.
  3. The DPO and PPO baselines are not mentioned to be “length aware” although length-aware training variants are quite common. This could give DCO a large advantage, as Figure 4 says DCO requires a carefully tuned truncation budget. At the very least, the experimental setting is not clear.
Questions for authors
  1. Are DPO and PPO length-aware?
  2. It seems like with greater compute budget, all methods will hit a perfect score/win-rate in Figure 2. Is this true?
Reviewer_fJTVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Dueling Constitutional Optimization (DCO), a derivative-free method for character training. The idea is to use a constitution to prompt an LLM judge, which compares the current model against a perturbed copy and returns a judgement: better, worse, or tie. The method keeps a Gaussian perturbation if the sum of the votes are positive. Both responses are truncated at a length budge (computed from the initial policy if i understand correctly), which the authors say acts as an implicit length penalty. The experiments use Gemma-3-4B and Qwen-3-4B on five traits, plus a composed persona, and compare against activation steering, online DPO and PPO with the same number of judge queries. The paper also gives a Hoeffding bound for the batch decision and an argument that cardinal judge scores are not identifiable.

Critical strengths
  1. The main strength is simplicity: DCO needs forward passes only, and the reported peak memory is 20.4 gb per gpu vs 55.9 gb for DPO and 62.1 gb for PPO on 4 H100's (Table A2). The implementations differ (custom vLLM vs OpenRLHF). The truncation keeps the response length flat without an explicit penalty. the k sweep in Fig. 4 shows where reward hacking starts. Ternary abstention also helps at small score gaps (Fig. 5b), though the end-to-end gain is pretty modest (0.92 vs 0.88 in Fig. 5a).

  2. The evaluation is broader than the training judge. The authors use a different judge for evaluation and check it against 15 human raters on 200 pairs (83% for GPT-5.4-mini and 77% for Qwen3.5-35B in Appendix D). Fig. 3 gives a length-controlled Bradley-Terry ranking with bootstrap intervals, and the raw head-to-head win rates are 0.72 against DPO and 0.76 against PPO. Appendix H provides stress test results on WildChat and four sycophancy probes. I also liked that the paper reports a negative result. Every persona method becomes more sycophantic (Table A5), and DCO is the least affected of the three trained methods.

Critical weaknesses
  1. The claim that DCO "consistently outperforms" the baselines is stronger than the evidence. The number of training runs is never stated, the bands in Figs. 2, 4 and A3 are undefined, and the Fig. 3 bootstrap resamples the 400 evaluation trials, so it does not capture run-to-run variation. That matters for a random search with about 88 accept/reject decisions per run. On Qwen, DCO's win rates are already 94-99% (Fig. 2b): DPO is level on Agreeable and Arrogant (95.6% vs 95.6% on both) and slightly ahead on Condescending (98.7% vs 98.1%). The Arrogant trait score is flat within a point, with DCO nominally highest on Qwen (DCO/DPO/PPO: 94.7/93.7/94.3; Gemma 71.8/71.2/71.8). So three of the five Qwen win-rate panels do not show the claimed advantage. The head-to-head comparison in Fig. 3 and the Gemma curves do favour DCO, so the method is not refuted, but Section 4.1 states more than the experiments show.

  2. There are also some protocol gaps that could change the comparison: the main-run k and L; whether the DPO and PPO judge calls are truncated the same way; how online DPO builds and labels its pairs and counts them toward the query budget; the selected DPO and PPO hyperparameters; and which conditions keep the Appendix K constitution prompt at generation time. The last one matters because, if the trained model is evaluated with the constitution in its system prompt, persona retention under "drop the persona" instructions partly measures instruction following, and the comparison with unprompted steering is confounded. Separately, every method's configuration is selected by GPT-5.4-mini win rate on training prompts (Appendix B); this is method-neutral, but the evaluation judge is not untouched by model selection.

  3. Section 5 analyzes the accept rule but never links it to progress on J_L, and its claims need narrowing. Section 3.1 says the search "actually optimizes" J_L, but Algorithm 1 accepts on the sign of a vote sum; with three prompts, (10, 10, 10) to (11, 11, 0) wins two duels and lowers the mean to 7.33. Proposition 1 fails when prompts repeat: with four rollouts per prompt, (0, 3) cannot get gap 1 while (1, 2) gets gap 2 under one increasing map, and Appendix E builds each map independently. Theorem 1 is tie-agnostic: the bound depends on the judge only through alpha = p_plus - p_minus, so a coin-flip tie-breaker has the same bound, and Hoeffding ignores the variance reduction from abstaining. The case for ternary feedback therefore rests on Fig. 5, not on the theorem.

Questions for authors
  1. Which k and L were used in the main runs? Were the DPO and PPO judge calls truncated in the same way? How are online DPO pairs generated, labelled, and counted toward the query budget? Which DPO and PPO configurations were selected?

  2. How many independent training runs are behind Figs. 2, A3, 3 and Table 1? What do the bands in Figs. 2, 4 and A3 represent? If several runs exist, could you report the per-run endpoints? If not, a few seeds on a subset of traits would help.

  3. Under what assumptions does a positive batch vote imply an increase in Eq. 3? If those assumptions need cross-prompt calibration, how does that square with Prop. 1? Should Prop. 1 be restricted to batches with distinct prompts?

  4. Fig. A2 reports a five-trait mean of about 95 at alpha = 0.2 on Gemma, but Table A9 shows Forgiving never above 49.9 at alpha = 0.2, and the five per-trait maxima average 88.1, while the AVG column peaks at 86.6. Which layers went into Fig. A2, and was alpha chosen by win rate (Section 4.1) or by trait score (Appendix J.2)?

Reviewer_yFGTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new method for character training of LLMs i.e. improving models to follow a given style/constitution. The authors argue that reinforcement learning with reward functions can be replaced with a simpler mechanism -- randomly perturb the weights of a pretrained model and use an comparison function to evaluate the improvement in instruction following over a batch of dataset. The authors show theoretically that the convergence of this process is proportional to the batch size of evaluation and error margin of the evaluator. It is a very interesting idea which empirically outperforms DPO and PPO on 4B size models. The authors evaluate the baselines and their methods on a benchmark introduced in the paper, and show that DCO leads to improved performance on a held out set of prompts. It also uses slightly lesser memory and training time. The main concerns I have are around the scalability of the method to larger size models, generalization to unseen personas, and the lack of testing on diverse benchmarks.

Critical strengths
  • The paper is easy to follow and well written. The authors do a good job of introducing the problem and intuitively explain their method and contribution.
  • The paper includes theory establishing the probability of improvement with random perturbations is a function of the batch size used to evaluate a candidate, the error margin in the evaluator and the rate of improvement of the sampler.
  • The authors provide details of their training setup, hyper parameter selection (including baselines), and ablations over different feedback setups and the affect of perturbation width to the training dynamics.
  • The base model capabilities does not degrade over basic math and reasoning problems.
Critical weaknesses
  • The current method is tested on a single benchmark with smaller models. Evaluating the method on at least one additional benchmark, discussion on how to scale this method and what are its implications, qualitative analysis and evidence of the improvement is important to understand the contribution.

  • The asymmetry between the model being trained and evaluator is not discussed i.e. while the training model is a 4B model, a much larger model is being used for evaluation. So, the process benefits from the generalization capabilities and better performance of a large model. What if one used the same model for evaluation (either both 4B or both larger ones)?

  • There is an important baseline that is missing i.e. how well do frontier models (say GPT-5.4 mini, which the authors use for evaluation anyways) perform when naively conditioned on the constitution perform? This gives us insights on the performance of the current models and understand is the capability lacking only in smaller models? Would be a very strong contribution if the current method actually outperforms frontier models at this task.

  • The authors should include a bigger discussion of the dataset introduced in the main paper. This is an important part of the contribution. Also a comment on why publicly available benchmarks for character training (say, PersonaBench) where not used?

  • It is unclear in the paper, if the model is being trained per-character or is a single model trained to follow different characters conditioned on the constitution? Apologies if I missed this in the paper.

Questions for authors
  • A missing citation is Gan et al. which introduces the idea that larger models are more easily optimized via random perturbations. A discussion around this and scalability of the method to larger models would make the contribution much stronger.

  • Would recommend the authors to clearly explain the data budget, and how is it normalized across the different baselines?

  • I am curious: under the noisy evaluator function why is DCO so much better. My first insight would be to expect similar asymptotic performance between GRPO and any other sampling based optimization method. Does DCO induce exploration than on-policy sampling cannot? or is there some regularization introduced which improves generalization?

  • How would you avoid reward hacking in this setting? Here the evaluator is a language model with its own biase, so the model is trained to align with "how well the evaluator understands the constitution"? Any discussion around this limitation and safety analysis would be interesting.

  • An interesting idea around using an evaluator with calibrated responses is to use typed-decision models (like Jev). This is just a recommendation and doesn't not affect my evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.