- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper trains only a small scoring head on a frozen language model, learning correctness and coherence coordinates from preference pairs and named rewrites. Precise one-factor edits in synthetic data can align the axes with the generating factors. On HelpSteer2, even when the rewrite directions are fitted perfectly, these coordinates do not become better evaluation criteria than those from preference-only training.
Critical strengths
- The problem is clearly formulated, and the theoretical claims stay within the stated assumptions. The paper shows that the Bradley–Terry preference likelihood is invariant under invertible coordinate transformations, and it gives a complete linear-algebra proof under explicit conditions, including a shared invertible mixing matrix and exact one-factor edits.
- The synthetic experiments separate fitting the total score from aligning the coordinates. The failure on real data is also supported by corresponding evidence.
Critical weaknesses
- The paper does not show that its rubric is useful. The experiments never use the learned coordinates to train a language model or replace human ratings, and the paper does not optimize a policy. Beyond the reported diagnostic comparisons, there is no evidence for what this rubric can actually be used for, which makes the contribution less compelling.
- The conditions required by the theory do not match the real training setup, and the HelpSteer2 results do not show that the method yields better criteria. The theory proves uniqueness under exact one-factor edits, which requires a shared invertible mixing matrix and coordinate changes confined to the named axis. Silver-200 fits its training-edit directions at 100%, but its rubric composition is 61.40%, below the 72.57% of preference-only training with development-set alignment. Real training therefore does not establish a more accurate correctness or coherence coordinate.
- I suggest that the authors tighten the main narrative to reduce obstacles to reading. The exposition is cumbersome. The core comparison is separated from the reader by protocol details and by metrics that should not be compared directly, including rubric composition, purity, and natural target effect.
Questions for authors
Refer to Weakness.