- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies pairwise human preference prediction for code generation outputs. It argues that the pointwise scoring paradigm commonly used in rubric-based LLM judges is not well suited to pairwise preference evaluation because it independently evaluates candidates A and B with respect to each criterion before comparing their scores. The paper further shows empirically that such rubric-based approaches can even underperform a monolithic judge.
To address this issue, the paper proposes CRITERALIGN. Whereas existing rubric-based approaches independently score each candidate for each criterion and then compare the resulting scores, CRITERALIGN directly compares the two candidates under each criterion. For criteria on which the two candidates are judged to be tied, the method introduces Batched Tie-Driven Criterion Refinement (BTCR), which decomposes the criterion into more fine-grained subcriteria. In addition, to mitigate position bias, a well-known source of bias in LLM judging, the paper applies Swap-Consistency Criterion Filtering (SCF) to improve consistency at the criterion level. Finally, the paper proposes Human-Preference-Aligned Guidance (HPAG), which summarizes recurring discrepancies between human preferences and the predictions and rationales of a monolithic judge into natural-language guidance. HPAG is injected into all three stages of the framework, namely criterion generation, criterion judging, and final judging, to better align LLM judgments with human preferences.
On the BigCodeReward validation set, CRITERALIGN improves the accuracy of the Qwen2.5-VL-32B judge by 5.9 percentage points over the monolithic baseline. The method also improves over matched monolithic baselines with Kimi-VL-A3B and Gemma-4-31B, suggesting some generalization across judge backbones and guidance synthesizers.
Critical strengths
The paper clearly identifies a limitation of existing pointwise rubric-based judging.
The paper shows that existing rubric-based approaches for pairwise judging can perform even worse than a simple monolithic judge. In particular, replacing pointwise RRD with pairwise criterion judging alone improves accuracy from 55.0% to 61.8%. This provides empirical evidence that, for pairwise preference tasks, it is important to generate criterion-level evidence in a directly comparative manner rather than independently evaluating the two candidates under each criterion and comparing the resulting scores afterward. This finding provides a useful insight beyond simply generating better or more numerous criteria. It highlights that how the criteria are used to compare candidates can itself be a critical design choice, and this insight could be useful for future rubric-based preference judging methods.
Critical weaknesses
1. Limited cross-model and cross-dataset validation of the main comparative finding.
The paper's central empirical claim goes beyond showing that CRITERALIGN outperforms a monolithic judge. A more important claim is that existing pointwise criterion-generation methods can underperform monolithic judges on pairwise preference prediction, and that reformulating criterion-level judging as direct pairwise comparison is important. However, the experiments in Table 1 that jointly compare existing criterion-generation baselines, the monolithic judge, and CRITERALIGN are conducted only with Qwen2.5-VL-32B as the judge backbone. In particular, one of the paper's most important findings, the improvement from Pointwise RRD to Pairwise RRD, is demonstrated only with this single backbone. It is therefore difficult to determine whether the degradation of pointwise criterion-based methods and CRITERALIGN's advantage over existing rubric-based methods persist across different judge backbones. Table 4 shows that CRITERALIGN improves over matched monolithic baselines for Kimi-VL-A3B, Gemma-4-31B, and Qwen2.5-VL-32B, but this comparison is limited to CRITERALIGN versus the monolithic baseline and does not establish whether the central pointwise-to-pairwise finding generalizes across models. It also does not explore factors such as model scale across these model families.
The evaluation is also largely concentrated on BigCodeReward. The appendix shows that HPAG synthesized on BigCodeReward transfers to WebDevJudge, but this is specifically a cross-dataset transfer experiment for HPAG rather than an evaluation of the full CRITERALIGN framework.
Overall, the current experiments show that CRITERALIGN improves over monolithic baselines across several judge backbones and that HPAG can transfer across datasets. However, they do not sufficiently establish whether the paper's central comparative finding generalizes beyond the specific combination of Qwen2.5-VL-32B and BigCodeReward. This limitation makes it difficult to determine whether the finding should be interpreted as a general principle of criterion-based judging or as an observation specific to the current experimental setting.
2. The contribution of human-preference grounding in HPAG is unclear.
The core motivation of HPAG is to capture the fact that human annotators and monolithic judges may emphasize different evaluation dimensions and to distill these human–judge rationale gaps into natural-language guidance. However, the human-side rationales used by HPAG are not explanations directly written by human annotators. Instead, an LLM is given the human overall preference and aspect-level votes and asked to retrospectively reconstruct why the human annotator may have preferred one candidate.
More specifically, HPAG compares the monolithic judge's prediction and rationale against this reconstructed human rationale, extracts recurring gaps in what the judge misses or over- and under-emphasizes, and summarizes these gaps into guidance. The concern is that the current experiments do not isolate whether this explicit rationale comparison is actually necessary. Because the reconstructed human rationale is already conditioned on human preference labels, it remains unclear whether HPAG's gains arise from identifying a genuine human–judge reasoning gap or simply from converting human preference labels or recurring judge failure patterns into natural-language instructions.
For example, the paper does not include simpler controls that generate guidance directly from human overall/aspect preference labels without reconstructing human rationales, or that summarize only human–judge disagreement cases into failure-driven guidance. The current results therefore establish that the final HPAG procedure is effective, but do not demonstrate that the proposed rationale-gap construction provides additional value beyond simpler label-based preference guidance or failure-driven prompting.
Questions for authors
1. Cross-model generalization
The paper's main comparison among Monolithic, Pointwise RRD, Pairwise RRD, and CRITERALIGN is conducted only with Qwen2.5-VL-32B. Could the authors provide the same comparison for Kimi-VL-A3B and Gemma-4-31B, including both Pointwise RRD and Pairwise RRD? In particular, I would be interested to know whether the improvement from pointwise to pairwise criterion judging, as well as the degradation of existing pointwise criterion-based methods relative to monolithic judging, can be reproduced across different judge backbones.
2. Cross-dataset generalization
The WebDevJudge experiment in the supplementary material shows that HPAG synthesized from BigCodeReward can transfer to another dataset, but it does not evaluate the full CRITERALIGN framework or compare Pointwise RRD and Pairwise RRD. Do the authors have results evaluating full CRITERALIGN together with Pointwise RRD and Pairwise RRD on WebDevJudge or another pairwise code-preference dataset? Such a comparison would help determine whether the pointwise-to-pairwise advantage and the relative gains of CRITERALIGN observed on BigCodeReward are dataset-specific.
3. Contribution of human rationale reconstruction in HPAG
How much of HPAG's performance gain comes specifically from the explicit comparison between reconstructed human rationales and judge rationales? The current ablations analyze the injection stages of HPAG and the effects of global versus category-level guidance, but they do not isolate the effect of the supervision signal used to construct HPAG. In particular, using the same training examples and guidance synthesizer, have the authors compared full HPAG against simpler alternatives such as (i) directly generating natural-language guidance from human overall/aspect preference labels without reconstructing human rationales, (ii) generating guidance only from disagreement patterns between human labels and monolithic judge predictions? Such comparisons would be important for separating the benefit of human preference supervision or judge failure information itself from the additional benefit provided by the paper's proposed reconstructed human–judge rationale gap.