- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a multi-agent framework called “Rubrics on Trial,” which evolves scoring criteria for specific queries by comparing pairs of synthetic responses—including alternative responses designed to identify unnecessary evaluation criteria. Experimental results show that, compared to several existing scoring criteria generation methods, this approach achieves improvements in both preference evaluation based on scoring criteria and downstream RLHF performance.
Critical strengths
-
The proposed synthetic response comparison provides an interesting approach to evaluating rubric quality without repeatedly training separate policies.
-
The experiments cover multiple preference benchmarks and downstream RLHF tasks. The ablation studies demonstrate the contributions of iterative evolution and both synthetic response pairs, with consistent improvements over several baselines.
Critical weaknesses
-
Downstream RLHF performance remains relatively weak in several settings (e.g., 20.56% on AlpacaEval 2.0 LC for Qwen2.5-3B). The paper also lacks comparisons with alternative preference optimization approaches such as DPO and SimPO, as well as recent self-improvement methods, making it difficult to assess the broader effectiveness of the proposed framework.
-
The method largely extends existing iterative rubric evolution approaches (e.g., RRD and ChaseTail) by incorporating synthetic response comparisons and additional criterion filtering. While these modifications improve performance, the underlying framework remains similar to prior work, and the paper does not sufficiently demonstrate a substantial methodological advance beyond existing rubric refinement techniques. For me, it's a lack of novelty.
Questions for authors
Why does the paper only compare downstream RLHF performance against rubric-generation methods? Could the authors include comparisons with DPO, SimPO, or recent self-improvement approaches under comparable data and computational budgets to better establish the practical effectiveness of the proposed framework?