acceptodds
Under review as a conference paper at ICLR 2027

CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging

Abstract

Pairwise human preference prediction is central to evaluating code-generation systems, where quality often depends on task-specific trade-offs beyond functional correctness. While rubric-based LLM judges improve interpretability by decomposing evaluation into explicit criteria, most existing pipelines remain pointwise: they score each response independently and derive preferences by comparing aggregated scores. We show that this design is poorly matched to pairwise code preference prediction and can underperform a strong monolithic judge. We propose CriterAlign, a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation through direct criterion-level pairwise judgments, tie-driven criterion refinement, swap-consistency filtering, and final pairwise synthesis. We further introduce Human-Preference-Aligned Guidance (HPAG), synthesized offline from training examples by extracting recurring rationale gaps between human preferences and monolithic judge predictions, and injected into the criterion generator, criterion judge, and final judge. On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy, with ablations confirming the contributions of pairwise criterion design and HPAG.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, LLM-as-a-judge, human preference, code evaluation, code preference, criterion-centric framework

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jM15OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies pairwise human preference prediction for code generation outputs. It argues that the pointwise scoring paradigm commonly used in rubric-based LLM judges is not well suited to pairwise preference evaluation because it independently evaluates candidates A and B with respect to each criterion before comparing their scores. The paper further shows empirically that such rubric-based approaches can even underperform a monolithic judge.

To address this issue, the paper proposes CRITERALIGN. Whereas existing rubric-based approaches independently score each candidate for each criterion and then compare the resulting scores, CRITERALIGN directly compares the two candidates under each criterion. For criteria on which the two candidates are judged to be tied, the method introduces Batched Tie-Driven Criterion Refinement (BTCR), which decomposes the criterion into more fine-grained subcriteria. In addition, to mitigate position bias, a well-known source of bias in LLM judging, the paper applies Swap-Consistency Criterion Filtering (SCF) to improve consistency at the criterion level. Finally, the paper proposes Human-Preference-Aligned Guidance (HPAG), which summarizes recurring discrepancies between human preferences and the predictions and rationales of a monolithic judge into natural-language guidance. HPAG is injected into all three stages of the framework, namely criterion generation, criterion judging, and final judging, to better align LLM judgments with human preferences.

On the BigCodeReward validation set, CRITERALIGN improves the accuracy of the Qwen2.5-VL-32B judge by 5.9 percentage points over the monolithic baseline. The method also improves over matched monolithic baselines with Kimi-VL-A3B and Gemma-4-31B, suggesting some generalization across judge backbones and guidance synthesizers.

Critical strengths

The paper clearly identifies a limitation of existing pointwise rubric-based judging.

The paper shows that existing rubric-based approaches for pairwise judging can perform even worse than a simple monolithic judge. In particular, replacing pointwise RRD with pairwise criterion judging alone improves accuracy from 55.0% to 61.8%. This provides empirical evidence that, for pairwise preference tasks, it is important to generate criterion-level evidence in a directly comparative manner rather than independently evaluating the two candidates under each criterion and comparing the resulting scores afterward. This finding provides a useful insight beyond simply generating better or more numerous criteria. It highlights that how the criteria are used to compare candidates can itself be a critical design choice, and this insight could be useful for future rubric-based preference judging methods.

Critical weaknesses

1. Limited cross-model and cross-dataset validation of the main comparative finding.

The paper's central empirical claim goes beyond showing that CRITERALIGN outperforms a monolithic judge. A more important claim is that existing pointwise criterion-generation methods can underperform monolithic judges on pairwise preference prediction, and that reformulating criterion-level judging as direct pairwise comparison is important. However, the experiments in Table 1 that jointly compare existing criterion-generation baselines, the monolithic judge, and CRITERALIGN are conducted only with Qwen2.5-VL-32B as the judge backbone. In particular, one of the paper's most important findings, the improvement from Pointwise RRD to Pairwise RRD, is demonstrated only with this single backbone. It is therefore difficult to determine whether the degradation of pointwise criterion-based methods and CRITERALIGN's advantage over existing rubric-based methods persist across different judge backbones. Table 4 shows that CRITERALIGN improves over matched monolithic baselines for Kimi-VL-A3B, Gemma-4-31B, and Qwen2.5-VL-32B, but this comparison is limited to CRITERALIGN versus the monolithic baseline and does not establish whether the central pointwise-to-pairwise finding generalizes across models. It also does not explore factors such as model scale across these model families.

The evaluation is also largely concentrated on BigCodeReward. The appendix shows that HPAG synthesized on BigCodeReward transfers to WebDevJudge, but this is specifically a cross-dataset transfer experiment for HPAG rather than an evaluation of the full CRITERALIGN framework.

Overall, the current experiments show that CRITERALIGN improves over monolithic baselines across several judge backbones and that HPAG can transfer across datasets. However, they do not sufficiently establish whether the paper's central comparative finding generalizes beyond the specific combination of Qwen2.5-VL-32B and BigCodeReward. This limitation makes it difficult to determine whether the finding should be interpreted as a general principle of criterion-based judging or as an observation specific to the current experimental setting.

2. The contribution of human-preference grounding in HPAG is unclear.

The core motivation of HPAG is to capture the fact that human annotators and monolithic judges may emphasize different evaluation dimensions and to distill these human–judge rationale gaps into natural-language guidance. However, the human-side rationales used by HPAG are not explanations directly written by human annotators. Instead, an LLM is given the human overall preference and aspect-level votes and asked to retrospectively reconstruct why the human annotator may have preferred one candidate.

More specifically, HPAG compares the monolithic judge's prediction and rationale against this reconstructed human rationale, extracts recurring gaps in what the judge misses or over- and under-emphasizes, and summarizes these gaps into guidance. The concern is that the current experiments do not isolate whether this explicit rationale comparison is actually necessary. Because the reconstructed human rationale is already conditioned on human preference labels, it remains unclear whether HPAG's gains arise from identifying a genuine human–judge reasoning gap or simply from converting human preference labels or recurring judge failure patterns into natural-language instructions.

For example, the paper does not include simpler controls that generate guidance directly from human overall/aspect preference labels without reconstructing human rationales, or that summarize only human–judge disagreement cases into failure-driven guidance. The current results therefore establish that the final HPAG procedure is effective, but do not demonstrate that the proposed rationale-gap construction provides additional value beyond simpler label-based preference guidance or failure-driven prompting.

Questions for authors

1. Cross-model generalization

The paper's main comparison among Monolithic, Pointwise RRD, Pairwise RRD, and CRITERALIGN is conducted only with Qwen2.5-VL-32B. Could the authors provide the same comparison for Kimi-VL-A3B and Gemma-4-31B, including both Pointwise RRD and Pairwise RRD? In particular, I would be interested to know whether the improvement from pointwise to pairwise criterion judging, as well as the degradation of existing pointwise criterion-based methods relative to monolithic judging, can be reproduced across different judge backbones.

2. Cross-dataset generalization

The WebDevJudge experiment in the supplementary material shows that HPAG synthesized from BigCodeReward can transfer to another dataset, but it does not evaluate the full CRITERALIGN framework or compare Pointwise RRD and Pairwise RRD. Do the authors have results evaluating full CRITERALIGN together with Pointwise RRD and Pairwise RRD on WebDevJudge or another pairwise code-preference dataset? Such a comparison would help determine whether the pointwise-to-pairwise advantage and the relative gains of CRITERALIGN observed on BigCodeReward are dataset-specific.

3. Contribution of human rationale reconstruction in HPAG

How much of HPAG's performance gain comes specifically from the explicit comparison between reconstructed human rationales and judge rationales? The current ablations analyze the injection stages of HPAG and the effects of global versus category-level guidance, but they do not isolate the effect of the supervision signal used to construct HPAG. In particular, using the same training examples and guidance synthesizer, have the authors compared full HPAG against simpler alternatives such as (i) directly generating natural-language guidance from human overall/aspect preference labels without reconstructing human rationales, (ii) generating guidance only from disagreement patterns between human labels and monolithic judge predictions? Such comparisons would be important for separating the benefit of human preference supervision or judge failure information itself from the additional benefit provided by the paper's proposed reconstructed human–judge rationale gap.

Reviewer_m7eCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve the accuracy of LLM-as-Judge in human pairwise preference under code generation scenarios, and propose a pipeline called CriterAlign, which includes tie-driven refinement (BTCR), swap-consistency filtering (SCF) and human-preference-aligned-guidance (HPAG) to improve scoring-based approach, and show some improvements with Qwen2.5-VL-32B on preference prediction accuracy on BigCodeReward benchmark.

Critical strengths

CriterAlign shows an interesting angle to convert rubric scoring to be multiple facets, and I would specifically list the following strengths:

  1. The motivation of this work makes sense and has potential to have real world impact: I would position this work as a practical and intuitive approach as LLM-as-Judge indeed can benefit from diversified preference criteria, and having specialized preference criteria can def bring benefit into pairwise comparison. This rescue the work from clear rejection, but not yet reach acceptance.

  2. Extensive and reasonable experiments and ablations: The authors put effort in designing extensive (and reasonable rather than defensive/not necessary) experiments to show the effectiveness of each component in their pipeline, and they show positive gain/certain level of generalizability (when using 2 additional models different than Qwen2.5-VL-32B in Table 4), which can practically justify the effectiveness of this approach. This also validate the result of the work.

Critical weaknesses

As mentioned in the Strengths, this paper provides an intuitive and practical angle to differentiate the utility of LLM-as-judge, but this practicality also means the method itself has limitations:

  1. Some reported gains are engineered and (some) are conditioned : The work to me is a little bit engineered, as multiple stacking components are not leading to steady gains (i.e., monolithic + HPAG has the highest proportional gain, and the other components have minor gains), therefore it's difficult to judge if these components are really contributing. Even for HPAG, it's actually guided by human preference labels rather than rationales, so it's questionable if there is alignment on human rationale, or just gain of labeling. The engineering part is the main issue that I cannot give weak acceptance.

  2. Current single benchmark result is not fully convincible: I appreciate that the authors took effort in designing different experimental setup (i.e., sampling 20% for synthesizing HPAG, etc), but more benchmarks should be necessary to prove the generalizability of this approach; furthermore, when cost/efficiency analysis is missing, the trade-off is unknown too. This can weaken the claims of this work. Similarly this is another reason that I cannot give weak acceptance.

Questions for authors
  1. If the monolithic (+HPAG) approach has achieved substantial gain and the full CriterionAlign (i.e., adding BTCR, etc) can only bring minor improvement, what's the motivation to include the full pipeline rather than dig a bit more into the mechanism of the monolithic approach? What's the accuracy-efficiency trade-off?

  2. Any additional experiments that can show improvement over baseline in different scenarios? BigCodeReward itself may not establish faithful conclusion of all the gains, regardless of how extensive the experiments are.

  3. Any justification of additional human rationales, when it can potentially introduce more randomness and bias into the pipeline? I would appreciate if there is any verification methods already in place that I may have missed during review to make sure those explanations are aligned with annotators judging reason.

If these are well answered I may raise my score or revise the comments accordingly.

Reviewer_BiaEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CriterAlign adapts rubric-based code evaluation to pairwise human preference prediction. It generates task-specific criteria, produces criterion-level A/B/tie/insufficient verdicts, refines tied criteria, filters verdicts that change under candidate swapping, and uses an LLM for final synthesis. HPAG is constructed offline from training examples and injected into multiple stages.

Critical strengths

Improvements are reported across three judge backbones.

Critical weaknesses
  1. The experiments do not isolate the contribution of the criterion pipeline

Whether the extra architecture contributes enough beyond that control to justify the method and its complexity. The largest component gain also changes more than one factor. I cannot verify robustness or the small component gains.

  1. The distinctive HPAG construction is insufficiently validated and documented

HPAG is presented as extracting recurring rationale gaps between human preferences and judge predictions. However, the human-side rationale is generated by another LLM conditioned on the example and human labels. The current controls do not distinguish the benefit of analyzing human–judge rationale gaps from ordinary preference-to-guidance distillation.

The synthesis inputs, the exact resulting principles, and the prompt differences across controls cannot be inspected. These omissions make it difficult to reproduce or resolve the attribution questions above.

Questions for authors

See Above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.