- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DCO as a technique to learn from model constitutions. DCO uses Noisy Comparison Random Search to first perturb model weights, rollout from the perturbed and original weights, and then compare the responses with a constitution-informed judge. If the judge accepts the rollouts from the perturbed weights more often than the original weights, the perturbation is kept. They show that this approach can beat DPO and PPO based approaches for constitutional AI. In addition, they also provide some theoretical analysis of batch decision reliability, and an explanation of why they use ordinal feedback instead of scalar score feedback.
Critical strengths
- The improvements over DPO and PPO are large.
- The algorithm is simple but effective in this application area.
- The presentation of the results is quite nice.
Critical weaknesses
- The main weakness to me is the level of novelty. This paper is an application of Noisy Comparison Random Search applied to constitutional AI style optimization (with some small tweaks like the truncation operation). To me this seems like the application of an already existing optimization approach to a well known problem, and I do not see it as a significant contribution to the community.
- The tasks and evaluation benchmarks are all author designed. As the methodology is not novel, the only contribution I could see are strong empirical results. The empirical results are strong, but they are on author designed benchmarks which makes it hard to evaluate their significance.
- The DPO and PPO baselines are not mentioned to be “length aware” although length-aware training variants are quite common. This could give DCO a large advantage, as Figure 4 says DCO requires a carefully tuned truncation budget. At the very least, the experimental setting is not clear.
Questions for authors
- Are DPO and PPO length-aware?
- It seems like with greater compute budget, all methods will hit a perfect score/win-rate in Figure 2. Is this true?