acceptodds
Under review as a conference paper at ICLR 2027

From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

Abstract

Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement. Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS-Hybrid provide effective signals for model optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Rubrics, Automatic Rubric Generation, Mechanistic Analysis, LLM-as-a-Judge

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_BqgAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper conducts experiments across multiple preference evaluation benchmarks and examines the intrinsic quality of the generated rubrics and their applications in model distillation and reinforcement learning. ARTS-Hybrid achieves the highest accuracy on five of the six main evaluation settings, while the cross-judge experiments and ablation studies also demonstrate promising results. The paper presents a clear research direction and a broad experimental evaluation, although some aspects regarding the independent contributions of its components, the validity of the calibration metrics, and the reliability of certain experimental improvements remain open to discussion.

Critical strengths
  1. Relevant research problem. Rubric design directly affects the reliability of large language model evaluation. The paper investigates criterion organization, criterion-level effectiveness, and score aggregation, providing a systematic perspective on how different design choices influence evaluation performance.

  2. Methodological novelty. ARTS-Hybrid combines adaptive tree search with semantic-first, response-based calibration, considering both the semantic validity and empirical scoring behavior of candidate criteria during hierarchical rubric construction. The integration of structured generation and behavioral calibration is well motivated.

  3. Comprehensive experimental evaluation. The paper includes structural analyses, counterfactual violation detection, evaluation across multiple benchmarks, cross-judge experiments, and component ablations. It also investigates the use of generated rubrics in model distillation and reinforcement learning, providing empirical evidence from several perspectives.

Critical weaknesses
  1. The independent methodological novelty and contributions of key components are not sufficiently clear.

ARTS-Hybrid integrates hierarchical decomposition, semantic filtering, response-based calibration, and global refinement into a unified framework. However, hierarchical rubric construction and iterative refinement have already been explored in related works such as Q-World and RRD. The main methodological contribution appears to lie in combining adaptive tree search with semantic-first calibration, but the individual contributions of these components are not sufficiently disentangled.

Table 14 primarily examines the effects of semantic validation and response-based calibration, without separately analyzing adaptive node expansion, priority scheduling, or global refinement. In Table 1, Tree improves coverage over the call-matched Flat-Multi baseline by only 0.61 and 2.77 percentage points under the two judge models, respectively. Although these results indicate some advantages of hierarchical generation, it remains unclear to what extent the improvements arise from the tree structure itself rather than repeated candidate generation and exploration. The current evidence does not fully clarify the necessity of the individual design choices or their incremental value over existing approaches.

  1. The relationship between the calibration metrics and actual evaluation effectiveness is not sufficiently validated.

The paper measures discriminative power using score variance across different responses. However, higher variance does not necessarily indicate that the scores are aligned with actual response quality. For example, a criterion may assign lower scores to high-quality responses and higher scores to low-quality responses while still exhibiting high discriminative power. Similarly, the stability metric in Appendix Equations (16)–(17) compares score variances across judge models, but similar variances do not imply consistent response rankings. Two judges may produce opposite rankings while still receiving a high stability score.

These metrics directly influence candidate criterion selection in ARTS-Hybrid, making their validity closely related to the core methodology. In Table 3, Semantic + Calibration achieves an average pairwise accuracy of 49.93%, an improvement of 1.86 percentage points over Direct at 48.07%, but the overall accuracy remains close to 50%. The current results do not sufficiently establish whether the proposed discrimination and stability metrics reliably reflect the quality of actual preference judgments.

  1. The statistical reliability of some experimental improvements and their downstream effectiveness remain uncertain.

Table 5 shows that ARTS-Hybrid achieves the highest accuracy on five of the six main evaluation benchmarks. However, the improvements over the strongest competing baseline on each benchmark range from 0.58 to 2.41 percentage points, and the method does not achieve the best result on PreciseIF. Although the cross-judge experiments in the appendix show more substantial gains, the smaller differences in the main experiments are not accompanied by sufficient statistical uncertainty analysis, leaving their reliability unclear.

The downstream improvements are also relatively limited. In Table 7, supervised fine-tuning with ARTS-Hybrid rubrics increases JudgeBench accuracy from 69.14% to 69.43%, a gain of only 0.29 percentage points. In Table 8, the reinforcement learning experiments improve performance over the base model by 0.79 and 0.32 points under the Thinking and Non-Thinking settings, respectively. These results suggest potential benefits of using the generated rubrics for model optimization, but the current evidence does not sufficiently establish the statistical reliability or practical significance of these relatively small improvements.

Questions for authors
  1. What are the main methodological contributions of ARTS-Hybrid compared with existing hierarchical rubric generation methods such as Q-World and RRD? What are the respective roles of adaptive tree search, node priority scheduling, and global refinement in the complete framework?

  2. Why does the paper use score variance to measure discriminative power rather than directly considering ranking consistency with response quality levels? How does the current stability metric account for cases where two judge models have identical score variances but produce opposite rankings?

  3. The paper constructs six calibration responses for each query, consisting of two strong, two medium, and two weak responses. What motivates this sample size and quality distribution? How sensitive are the calibration results to changes in the number and composition of calibration responses?

  4. Is statistical uncertainty analysis available for the relatively small improvements reported in Tables 5, 7, and 8? To what extent do these results support consistent improvements in evaluation performance and downstream model optimization?

6 comments
AuthorsOpenReview ↗

Response to Weakness 1& 2

Weakness 1: The independent methodological novelty and contributions of key components are not sufficiently clear.

Thank you for the suggestion. We will clarify that the paper’s main contribution is its analysis of rubric design mechanisms and the construction method derived from these findings. We do not claim that hierarchical decomposition or iterative refinement is itself new. We separately examine how criteria are organized, selected, and aggregated to explain which design choices help and under what conditions, rather than simply proposing a pipeline with higher end-to-end scores. This is also consistent with Reviewer 1kHW’s assessment of our contribution: “It tries to explain why rubrics work, not just show another method that scores higher.”

The methodological contribution of ARTS-Hybrid lies in translating these findings into adaptive search and behavioral calibration under semantic constraints. These components serve distinct roles; criteria are not selected solely for their ability to separate response scores.

To examine whether the benefits of tree construction arise from repeated generation and exploration, we added a Flat-Iterative control. Flat-Iterative reads the complete current criterion set at each round and adds missing criteria, splits overly broad criteria, and merges or removes duplicates. For each query, we match TREE’s total input/output token budget and final criterion count, including the cost of feeding the complete rubric back into the model at every iteration. We also report actual token usage and final rubric length. The results and actual construction costs are shown below:

MethodCoverage (%) ↑Novelty (%) ↑Mean input tokensMean output tokensMean criterion countMean rubric length (tokens)
TREE59.3410.577,364.021,653.049.72302.42
Flat-Iterative40.0010.716,240.672,394.449.72272.57

Across all samples, Flat-Iterative consumed 95.83% of TREE’s total construction tokens. On the 1,430 paired records where both methods completed successfully and satisfied the constraints, TREE achieved 59.34% Coverage, compared with 40.00% for Flat-Iterative.

These results show that TREE retains a clear coverage advantage with comparable actual token consumption and matched final criterion counts, even when the flat baseline can integrate and refine criteria across rounds, providing further support for hierarchical construction.

Weakness 2: The relationship between the calibration metrics and actual evaluation effectiveness is not sufficiently validated.

We agree with the distinction raised by the reviewer, which is also reflected in Table 3: higher score variance does not imply that responses are distinguished in the correct direction, and similar variances across judges do not imply consistent response rankings. We therefore do not treat these metrics as sufficient conditions for criterion validity. Instead, we use them to help compare candidate decompositions under semantic constraints.

Table 3 illustrates the limitations of relying on behavioral calibration alone. Calibration achieves the best discrimination and cross-judge variance metrics, yet 17.84% of its selected revisions violate the semantic validity constraints. This shows that optimizing scoring behavior can select criteria that deviate from task requirements, motivating our approach of semantic filtering followed by behavioral calibration. We therefore also evaluate preference judgments on independent response pairs.

MethodDisc. ↑Judge Var. ↓Invalid (%) ↓Llama Accuracy (%)Qwen Accuracy (%)Avg. Accuracy (%) ↑
Direct0.51050.2749—48.2747.8748.07
Semantic0.52070.27260.0049.8747.0048.44
Calibration0.54500.251017.8448.2749.7349.00
Semantic + Calibration0.52950.26260.0049.3350.5349.93

The near-50% accuracy in Table 3 should also be understood in the context of RubricBench’s design. The dataset specifically selects difficult response pairs involving complex requirements, misleading surface features, and reasoning errors. Table 2 of the original RubricBench paper also reports several overall accuracies below 50%:

MethodBackboneRubricBench Overall Accuracy (%)
Vanilla JudgeGPT-4o-mini40.2
Vanilla JudgeDeepSeek-v3.238.8
TICKGPT-4o-mini45.2
OpenRubricGPT-4o-mini46.7

These results show that performance near or below chance is not unique to our setting and reflects the challenges this benchmark poses for existing evaluation methods.

RQ2 therefore examines the complementary roles of semantic filtering and behavioral calibration, it does not claim to have solved reliable preference judgment on RubricBench. We will clarify this scope in the revised manuscript.

AuthorsOpenReview ↗

Response to Weakness 3

Weakness 3: The statistical reliability of some experimental improvements and their downstream effectiveness remain uncertain.

For the main evaluation, we performed 10,000 query-level paired bootstrap resamples, sampling all response pairs belonging to the same query together and using identical sampling indices for both methods. Compared with Q-World, ARTS-Hybrid improves accuracy by 8.10 percentage points on RB Chat (95% CI: [4.31, 12.02]) and 2.56 percentage points on Focus (95% CI: [0.40, 4.71]).

For distillation, Table 7 examines not only the evaluation performance of the rubrics themselves, but also whether they can serve as supervision for training a separate rubric generator. Under the same training data and settings, distillation using other rubric sources yields accuracies below the base generator’s 69.14%, while ARTS-Hybrid reaches 69.43%. This comparison suggests that the source of rubrics used for distillation can affect training outcomes, and generated rubrics do not automatically provide effective supervision. ARTS-Hybrid achieves the highest accuracy in this experiment while maintaining the base generator’s performance level.

To further assess the stability of the results, we report the mean accuracy and standard deviation across two runs, as shown below.

SFT SupervisionMean (%) ± SD (pp)
Direct-Prompt Rubrics66.58 ± 0.40
OpenRubrics Expert Rubrics67.43 ± 1.22
Q-World Rubrics66.29 ± 2.02
ARTS-Hybrid Rubrics69.29 ± 0.21

Similarly, the RL experiment examines whether rubrics from different sources are suitable reward signals for post-training. Table 8 shows that different rubric sources do not produce consistent training gains: Direct-Prompt performs slightly below the base model in both settings, while the effects of Q-World and expert rubrics vary across settings. ARTS-Hybrid achieves the highest score in both settings, exceeding the base model.

We provide two additional analyses. First, we conducted paired significance tests for the Thinking setting using the original Qwen2.5-7B-Instruct evaluator. The improvements over Base, Direct-Prompt, and Q-World are significant in the unadjusted tests; the improvement over Q-World remains significant after Holm correction.

ComparisonΔ (pp)Paired 95% CIUnadjusted pHolm-adjusted p
ARTS-Hybrid − Base model+0.79[+0.21, +1.37]0.008210.09852
ARTS-Hybrid − Direct-Prompt+0.84[+0.25, +1.43]0.004850.06305
ARTS-Hybrid − OpenRubrics Expert+0.59[−0.01, +1.20]0.055050.60554
ARTS-Hybrid − Q-World+1.28[+0.71, +1.87]0.000020.00028

Second, we re-evaluated the saved responses using the larger Qwen2.5-32B-Instruct evaluator. The Thinking-setting results are shown below:

Reward sourceQwen2.5-7B-InstructQwen2.5-32B-Instruct
Base model38.1457.26
ARTS-Hybrid38.9357.32

With Qwen2.5-32B-Instruct, ARTS-Hybrid retains a positive point estimate relative to the base model.

AuthorsOpenReview ↗

Response to Question 1

Question 1: What are the main methodological contributions of ARTS-Hybrid compared with existing hierarchical rubric generation methods such as Q-World and RRD? What are the respective roles of adaptive tree search, node priority scheduling, and global refinement in the complete framework?

The methodological contribution of ARTS-Hybrid lies in integrating semantic constraints and response-based behavioral calibration into the step-by-step construction of rubrics, rather than introducing hierarchical decomposition or iterative revision for the first time. Building on the structured construction and refinement explored by Q-World and RRD, we emphasize that candidate decompositions should be semantically sound and exhibit useful scoring behavior on actual responses. We therefore first construct candidate sets that satisfy semantic constraints, then use the calibration objective to select a decomposition, rather than relying solely on the semantic quality of the generated text. RQ1–RQ3 examine the roles of structure, criterion selection, and aggregation, respectively, providing the basis for this design.

The three components serve distinct roles within the framework:

ComponentRole
Adaptive tree searchDetermines which criteria require further decomposition, which set of child criteria to adopt, and when to stop. Atomic nodes that can be assessed independently remain leaves; other nodes are expanded within the depth and leaf budgets.
Node priority schedulingDetermines which node to expand next under a limited budget. It combines task importance and depth through P(v)=I(v)γdvP(v)=I(v)\gamma^{d_v} to prioritize exploration; it does not determine which candidate decomposition is better.
Global refinementReviews the complete criterion set after local construction to address missing requirements, cross-branch duplication, and non-atomic criteria. Passing local checks does not guarantee that the final set is globally complete and non-redundant.

Question 2: Why does the paper use score variance to measure discriminative power rather than directly considering ranking consistency with response quality levels? How does the current stability metric account for cases where two judge models have identical score variances but produce opposite rankings?

We use score variance to measure separation between responses, rather than explicitly requiring scores to follow the intended Strong–Medium–Weak ordering, for two reasons. First, these quality levels are controlled through generation prompts and do not constitute a verified ranking of individual responses. A response with higher overall quality may not perform better on every specific criterion. Second, ranking metrics do not capture score magnitudes: two judges may produce the same ranking while assigning different sizes to the gaps between responses. We therefore use variance as a proxy signal for behavioral calibration to help select candidates after semantic filtering, rather than as a guarantee that responses are distinguished in the correct direction.

Regarding identical variances with opposite rankings, we agree with the reviewer: the current variance-based stability metric cannot distinguish this case. More broadly, higher score variance does not imply correct discrimination, and agreement between judges does not guarantee valid judgments. We therefore do not treat variance or agreement as sufficient conditions for criterion validity, but use these signals to assist candidate selection under semantic constraints.

Table 3 illustrates a related limitation of relying on behavioral calibration alone. Calibration achieves the best discrimination and cross-judge variance metrics, yet 17.84% of its selected revisions violate the semantic validity constraints. This shows that optimizing scoring behavior can select criteria that deviate from task requirements, motivating ARTS-Hybrid’s approach of semantic filtering followed by behavioral calibration. Semantic + Calibration reduces this invalid-selection rate to zero while achieving the highest average pairwise accuracy.

MethodDisc. ↑Judge Var. ↓Invalid (%) ↓Llama Accuracy (%)Qwen Accuracy (%)Avg. Accuracy (%) ↑
Direct0.51050.2749—48.2747.8748.07
Semantic0.52070.27260.0049.8747.0048.44
Calibration0.54500.251017.8448.2749.7349.00
Semantic + Calibration0.52950.26260.0049.3350.5349.93
Reviewer_NFr4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies what makes LLM-generated rubrics effective. It analyzes three stages of rubric design separately: how criteria are organized, what makes individual criteria effective, and how criterion scores are aggregated. The analysis suggests that hierarchical organization improves criterion coverage, that semantic specification alone is not enough to make criteria discriminative, and that the choice of aggregation matters little. Based on these findings, the authors propose ARTS-Hybrid, which builds rubrics through adaptive tree search and selects criteria with a semantic-first, response-calibrated procedure. Experiments on several preference benchmarks show improved pairwise accuracy and rubric quality, and additional experiments examine rubric-generator distillation and rubric-based RL.

Critical strengths
  1. Disentangling the contributions of individual rubric design choices is a worthwhile question that prior work has not examined systematically.
  2. The controlled analyses are carefully designed, using call-matched flat baselines, minimally edited counterfactual responses, and a shuffled-path control.
  3. The released gold rubric trees and counterfactual responses are a useful resource, and the paper is clearly written.
Critical weaknesses
  1. The effects reported in the analysis are small and come without variance estimates or significance tests. In Table 3, all settings stay below 50% pairwise accuracy, so the conclusions of RQ2 rest on judgments that are no better than chance.
  2. ARTS-Hybrid has many hyperparameters, and some of them are reported as value sets with no selection protocol. The ablation removes only semantic validation and calibration. The contributions of tree search, global repair, weighting, and the individual terms of the objective remain unverified, and removing calibration actually improves results on Focus.
  3. The main gains are marginal on small benchmarks. On JudgeBench, ARTS-Hybrid beats Q-World by 2 examples and Direct Judge by 4, and the distillation gain in Table 7 is a single example. RubricBench appears in the experimental setup but is missing from Table 5.
  4. The evidence for downstream use is weak. In Table 8, ARTS-Hybrid rubrics improve HealthBench by only 0.79 and 0.32 points, and all rubric sources stay within one point of the untrained model, which seems to point to an ineffective RL setup rather than better rewards.
Questions for authors

Please check my concerns and problems in weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Weakness 1

Weakness 1A: The effects reported in the analysis are small and come without variance estimates or significance tests.

Thank you for pointing out the lack of significance tests. We added three sets of supplementary results to address this concern.

First, we supplemented the main comparisons in Tables 1 and 2 with statistical tests and confidence intervals. We used query-level paired bootstrap confidence intervals and permutation tests, with Holm correction across the comparisons within each table. Bold values indicate positive differences.

Table 1. Coverage and Novelty comparisons

EvaluatorMetricComparisonΔ (pp)95% CIHolm-adjusted p
DeepSeekCoverageTREE − FLAT-MULTI+3.11[1.72, 4.51]0.00018
DeepSeekCoverageTREE − FLAT+4.98[3.35, 6.66]0.00008
DeepSeekNoveltyTREE − FLAT-MULTI+2.06[1.05, 3.09]0.00056
DeepSeekNoveltyTREE − FLAT+1.97[0.68, 3.27]0.00630
LlamaCoverageTREE − FLAT-MULTI+2.55[1.37, 3.76]0.00020
LlamaCoverageTREE − FLAT+1.94[0.44, 3.42]0.01196
LlamaNoveltyTREE − FLAT-MULTI+2.62[1.18, 4.07]0.00120
LlamaNoveltyTREE − FLAT+4.79[2.92, 6.64]0.00008

Table 2. Counterfactual evaluation comparisons

EvaluatorCounterfactual typeComparisonΔ (pp)95% CIHolm-adjusted p
DeepSeekSingle-leafPATH − FLAT+3.94[2.07, 5.81]0.00036
DeepSeekSingle-leafPATH − TREE+4.21[2.27, 6.14]0.00036
DeepSeekWithin-dimensionPATH − FLAT+0.54[−0.88, 1.96]0.51070
DeepSeekWithin-dimensionPATH − TREE+1.48[0.07, 2.97]0.16956
DeepSeekCross-dimensionPATH − FLAT+4.26[2.84, 5.75]0.00012
DeepSeekCross-dimensionPATH − TREE+11.78[9.80, 13.77]0.00012
LlamaSingle-leafPATH − FLAT+3.34[1.34, 5.28]0.00396
LlamaSingle-leafPATH − TREE+5.01[2.94, 7.14]0.00020
LlamaWithin-dimensionPATH − FLAT+2.29[1.08, 3.57]0.00185
LlamaWithin-dimensionPATH − TREE+2.77[1.42, 4.12]0.00042
LlamaCross-dimensionPATH − FLAT+1.35[0.00, 2.70]0.16956
LlamaCross-dimensionPATH − TREE+3.48[1.85, 5.11]0.00036

Second, we performed three independent end-to-end runs of ARTS-Hybrid and Direct Judge on RB Chat to examine both their relative performance and variation across runs.

MethodRun 42 (%)Run 123 (%)Run 2026 (%)Mean (%) ± SD (pp)
Direct Judge86.0386.3186.3186.22 ± 0.16
ARTS-Hybrid90.5090.2288.2789.66 ± 1.22

Third, we repeated the shuffled-path control with two sampling/shuffling seeds:

EvaluatorCounterfactual typeSeed 1: CorrectSeed 1: ShuffledSeed 2: CorrectSeed 2: Shuffled
DeepSeek-V4-Flash-GASingle-leaf23.7723.63 (−0.13)21.2321.50 (+0.27)
DeepSeek-V4-Flash-GAWithin-dimension12.5511.61 (−0.94)12.2811.88 (−0.40)
DeepSeek-V4-Flash-GACross-dimension27.6625.67 (−1.99)27.0925.11 (−1.99)
Llama3.1-405B-InstructSingle-leaf35.9134.05 (−1.87)37.5236.98 (−0.53)
Llama3.1-405B-InstructWithin-dimension14.7113.09 (−1.62)15.6513.77 (−1.89)
Llama3.1-405B-InstructCross-dimension43.2642.41 (−0.85)44.2642.55 (−1.70)

All experimental results require us to reproduce and run the corresponding evaluation pipelines. Within the rebuttal period, we therefore prioritized statistical analyses of the key comparisons and the repeated runs reported above. Repeated experiments do not yet cover every setting, and we appreciate the reviewer’s understanding.

Weakness 1B: In Table 3, all settings stay below 50% pairwise accuracy, so the conclusions of RQ2 rest on judgments that are no better than chance.

The near-50% accuracy in Table 3 should be considered in the context of RubricBench’s task design. The benchmark selects difficult response pairs involving complex requirements, misleading surface features, and reasoning errors. Its original Table 2 also reports several overall accuracies below 50%:

MethodBackboneRubricBench Overall Accuracy (%)
Vanilla JudgeGPT-4o-mini40.2
Vanilla JudgeDeepSeek-v3.238.8
TICKGPT-4o-mini45.2
OpenRubricGPT-4o-mini46.7

These results show that near- or below-chance performance is not unique to our experiments and also reflects the difficulty of RubricBench for existing evaluators. They provide context for the low absolute scores, rather than evidence of our method’s effectiveness.

AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2A: ARTS-Hybrid has many hyperparameters, and some of them are reported as value sets with no selection protocol.

We apologize for not making the selection protocol clear. We used PreciseIF only for hyperparameter selection, then fixed a single configuration for evaluation on the remaining benchmarks. No hyperparameter tuning was performed on those remaining evaluation sets. We have updated Table 5 to identify PreciseIF as the configuration-selection dataset. The fixed configuration is listed below.

Table 11: The hyperparameter configuration used by ARTS-Hybrid.

HyperparameterValue
Leaf budget16
Candidates per expansion10
Minimum children per expansion3
Maximum tree depth3
Depth discount γ\gamma0.85
Calibration responses per query6
Calibration judges2
Semantic score threshold0.60
Calibration threshold τscor\tau_{\mathrm{scor}}0.80
Coverage threshold τcov\tau_{\mathrm{cov}}1.00
Maximum regeneration attempts4
Objective weights (αD,αA,αS)(\alpha_D,\alpha_A,\alpha_S)(0.25, 0.30, 0.45)
Redundancy penalty λred\lambda_{\mathrm{red}}0.05
Size penalty λsize\lambda_{\mathrm{size}}0.00
Maximum global repair rounds2

Weakness 2B: The ablation removes only semantic validation and calibration. The contributions of tree search, global repair, weighting, and the individual terms of the objective remain unverified, and removing calibration actually improves results on Focus.

We examine the benefits of tree structure and search in three parts, with additional experiments in the corresponding sections.

We added Flat-Iterative, which revises the complete flat rubric across rounds by adding missing criteria, splitting overly broad criteria, and removing duplicates. For each query, we match TREE’s total input/output token budget and final criterion count. This control helps distinguish the benefits of hierarchical construction from those of repeated generation and iterative refinement.

We therefore added Flat-Iterative: at each round, it reads the query and the current complete flat rubric, adds missing criteria, splits overly broad criteria, and merges or removes duplicates. This gives the flat baseline the opportunity to incorporate information from multiple rounds into its final rubric.

Because Flat-Iterative and TREE differ in the amount of information processed and the cost per call, we compare them under the same total token budget and final criterion-count constraint. The budget includes both input and output tokens, including the full rubric passed back in at each iteration. We also report actual input/output token usage, final criterion counts, and rubric lengths. The results and actual construction costs are shown below:

MethodCoverage (%) ↑Novelty (%) ↑Mean input tokensMean output tokensMean criterion countMean rubric length (tokens)
TREE59.3410.577,364.021,653.049.72302.42
Flat-Iterative40.0010.716,240.672,394.449.72272.57

Across all samples, Flat-Iterative consumed 95.83% of TREE’s total construction tokens. On the 1,430 paired records where both methods completed successfully and satisfied the constraints, TREE achieved 59.34% Coverage, compared with 40.00% for Flat-Iterative—a gain of 19.34 percentage points (95% CI: [17.57, 21.13]).

These results show that TREE retains a clear coverage advantage with comparable actual token consumption and matched final criterion counts, even when the flat baseline can integrate and refine criteria across rounds, providing further support for hierarchical construction.

For the individual objective terms, we added two ablations, removing SS and RR separately and rerunning the complete search. The results are shown below:

SettingJudgeBench Accuracy (%)
Full64.29
without_S61.43
without_R62.86

Regarding weighting, RQ3 compares different aggregation and weighting schemes, as shown below:

Aggregation SchemeDeepSeek-V4-Flash-GA (%) ↑Llama3.1-405B-Instruct (%) ↑
FLAT73.2073.00
TREE72.60 (−0.60)71.33 (−1.67)
FLAT + WU72.20 (−1.00)73.27 (+0.27)
TREE + WU72.53 (−0.67)71.33 (−1.67)
AuthorsOpenReview ↗

Response to Weakness 3

Weakness 3A: The main gains are marginal on small benchmarks. On JudgeBench, ARTS-Hybrid beats Q-World by 2 examples and Direct Judge by 4, and the distillation gain in Table 7 is a single example.

Our main contribution is the analysis of rubric design mechanisms and the construction method derived from those findings. We do not base this contribution on the small JudgeBench gain alone. This is also consistent with Reviewer 1kHW’s assessment of our contribution: “It tries to explain why rubrics work, not just show another method that scores higher.”

Although ARTS-Hybrid correctly judges only two additional examples compared with Q-World in Table 5, the size of the improvement varies across model settings. With Qwen3.5-397B-A17B and Llama3.1-405B-Instruct, ARTS-Hybrid also outperforms the baselines on JudgeBench, with larger numerical margins:

MethodQwen3.5-397B-A17B Accuracy (%)Llama3.1-405B-Instruct Accuracy (%)
Direct Judge77.1457.40
Query-Only80.5741.50
Response-Aware79.4356.00
Q-World80.0063.86
ARTS-Hybrid (ours)85.6769.71

The improvements over the strongest competing baseline are 5.10 percentage points with Qwen3.5-397B-A17B and 5.85 points with Llama3.1-405B-Instruct.

We also supplemented the GLM-4.7-Flash results with query-level paired bootstrap analyses using 10,000 resamples. All response pairs belonging to the same query were resampled together, preserving their dependence. Relative to Q-World, the results are:

BenchmarkARTS-Hybrid Accuracy (%)Q-World Accuracy (%)Δ (pp)Paired 95% CI
RB Chat89.3981.28+8.10[4.31, 12.02]
Focus85.5983.03+2.56[0.40, 4.71]

These results show that the observed benefits are not limited to a few additional correct judgments on JudgeBench. We will distinguish these findings from the smaller and less certain improvements, and avoid presenting every numerical gain as an established advantage.

For distillation, Table 7 examines not only the evaluation effectiveness of the rubrics themselves, but also whether they can serve as supervision for training a separate rubric generator. With the same training examples and training settings, distillation using the other rubric sources yields accuracies below the base generator’s 69.14%, whereas ARTS-Hybrid achieves 69.43%. This comparison suggests that the source of rubric supervision can affect training outcomes, and generated rubrics do not automatically provide effective supervision. In this experiment, ARTS-Hybrid achieves the highest accuracy while maintaining the base generator’s performance level.

Weakness 3B: RubricBench appears in the experimental setup but is missing from Table 5.

RubricBench is used in our mechanistic analysis. We also report its preference accuracy below:

MethodLlama Accuracy (%)Qwen Accuracy (%)Avg. Accuracy (%) ↑
Direct48.2747.8748.07
Semantic49.8747.0048.44
Calibration48.2749.7349.00
Semantic + Calibration (ours)49.3350.5349.93

These results address our mechanistic questions about semantic filtering and behavioral calibration. Including RubricBench in Table 5 would require reproducing all baselines—including Direct Judge, Query-Only, Response-Aware, and Q-World—under the same evaluation protocol, as we did for the other benchmarks. Given the limited rebuttal period, we prioritized the statistical analyses supporting the mechanistic experiments. We have clarified RubricBench’s role in the revised manuscript to avoid ambiguity between the experimental setup and the benchmarks reported in Table 5. We appreciate the reviewer’s understanding.

AuthorsOpenReview ↗

Response to Weakness 4

Weakness 4: The evidence for downstream use is weak.

The RL experiment examines whether rubrics from different sources are suitable reward signals for post-training. Table 8 shows that different rubric sources do not produce consistent training gains: Direct-Prompt performs slightly below the base model in both settings, while the effects of Q-World and expert rubrics vary across settings. ARTS-Hybrid achieves the highest score in both settings, exceeding the base model.

We provide two additional analyses. First, we conducted paired significance tests for the Thinking setting using the original Qwen2.5-7B-Instruct evaluator. The improvements over Base, Direct-Prompt, and Q-World are significant in the unadjusted tests; the improvement over Q-World remains significant after Holm correction.

ComparisonΔ (pp)Paired 95% CIUnadjusted pHolm-adjusted p
ARTS-Hybrid − Base model+0.79[+0.21, +1.37]0.008210.09852
ARTS-Hybrid − Direct-Prompt+0.84[+0.25, +1.43]0.004850.06305
ARTS-Hybrid − OpenRubrics Expert+0.59[−0.01, +1.20]0.055050.60554
ARTS-Hybrid − Q-World+1.28[+0.71, +1.87]0.000020.00028

To further examine these differences, we re-evaluated the saved answers using the larger Qwen2.5-32B-Instruct evaluator. The Thinking-mode results are shown below:

Reward sourceQwen2.5-7B-InstructQwen2.5-32B-Instruct
Base model38.1457.26
ARTS-Hybrid38.9357.32

Under Qwen2.5-32B-Instruct, ARTS-Hybrid retains a positive point estimate relative to the base model in Thinking mode. We will therefore soften our claims about transfer to policy optimization, presenting these experiments as exploratory evidence on the suitability of different rubric sources as reward signals.

We hope the additional experiments and clarifications address your concerns and strengthen the evidence supporting our conclusions. Thank you again for your constructive feedback. We would be happy to discuss any remaining questions.

Reviewer_1kHWOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Problem: Auto-generated rubrics for AI judges work, but it's unclear why.

Analysis (3 findings):

  1. Structure: Tree-organized rubrics cover more criteria. They help judges as context, not as top-down shortcuts.
  2. Criteria: Instructions alone don't make good criteria. Filter for sensible ones, then test them on sample answers.
  3. Aggregation: Score weighting barely matters.

Method: ARTS-Hybrid builds a rubric tree, filters candidate criteria for clarity and relevance, picks those that best separate strong from weak sample answers, then validates and weights the final set.

Results: Best on 5/6 benchmarks, higher-quality rubrics, and small gains when used as training rewards.

Critical strengths

1. It tries to explain why rubrics work, not just show another method that scores higher.

Most papers on rubric generation just report end-to-end accuracy, so it's hard to tell which design choices actually help. Here the authors look at three things separately (how criteria are organized, how individual criteria are written and selected, and how scores are combined) and use reasonable controls for each. The FLAT-MULTI baseline with matched generator calls, the shuffled-path check, and the counterfactual responses with isolation checks were all good choices. This is the main reason I lean positive: the analysis gives some real guidance on rubric design, and the proposed method follows from it rather than being one more pipeline.

2. Some of the findings are useful in practice and not entirely obvious.

  • Using the hierarchy as context for the judge (PATH) helps, but using it to prune branches top-down (TREE) hurts.
  • Picking criteria only by how they score on sample responses lets through a fair number of invalid ones (about 18%), so a semantic filter is needed first.
  • Weighting and aggregation schemes make little difference.

These are things people building rubric-based judges or reward models could apply directly, even without adopting ARTS-Hybrid. The gains of the method itself are modest, so for me these findings carry more of the paper's value. The released gold rubric trees and counterfactual data are also helpful for follow-up work.

Critical weaknesses

1. Most of the evidence comes from small differences with no measure of variance.

A lot of the conclusions rest on gaps of 0.5 to 3 points: tree vs. FLAT-MULTI coverage (57.39 vs. 56.78 with DeepSeek), PATH vs. FLAT, the shuffled-path control (often under 1 point), and the aggregation comparison. There are no error bars, repeated runs, or significance tests anywhere, so I can't tell which of these are real. The RQ2 results are the most worrying. All four settings in Table 3 land around 48–50% pairwise accuracy, which is close to chance, so it's hard to say that semantic filtering plus calibration "works best" when nothing really works on that set. Since the paper's main pitch is the analysis, this matters a lot: if the findings aren't clearly supported, the method built on them loses much of its motivation.

2. The comparison with baselines isn't on equal footing, and the downstream claims go beyond the results.

ARTS-Hybrid gets six extra calibration responses from three other models plus two extra judge models, and it makes many more LLM calls per query. The baselines get none of this, and the paper doesn't report compute cost or try a matched-budget baseline. Closely related methods discussed in the related work (RRD, RubricHub, the concurrent tree-based method) are also missing from the comparison. On top of that, the transfer claims are stronger than the numbers:

  • Distillation: the gain is 69.43 vs. 69.14 for the untrained model.
  • RL: the gain is under 1 point on HealthBench, from a single run judged by a 7B model.

Taken together, it's hard to know whether the modest improvements come from the method itself or just from the extra information and compute.

Questions for authors
  1. Can you report variance across multiple runs or seeds, or significance tests, for the main comparisons (Tables 1–5)? Many of the gaps are 1–3 points, and it's hard to judge them without some sense of noise.

  2. In Table 3, all settings are around 48–50% pairwise accuracy, which is close to chance. Why is performance so low on this subset compared to the other benchmarks? Do the RQ2 conclusions still hold in a setting where the rubrics actually separate responses well?

  3. How many LLM calls (or how much compute) does ARTS-Hybrid need per query compared to Q-World and the one-shot baselines? Have you tried giving the baselines a matched budget, or access to the same calibration responses?

  4. The calibration panel uses one model family per quality tier (Qwen = strong, Llama = medium, Mistral = weak). How do you rule out that calibration is selecting criteria that pick up model style rather than response quality? Did you try mixing tiers across models?

  5. Why are RRD, RubricHub, and the concurrent tree-based method (Yang et al., 2026) not included as baselines? Also, RubricBench is listed in the setup but I couldn't find its results in Table 5.

  6. Several hyperparameters are given as sets (e.g., candidates per expansion {6, 10}, τ_cov {0.80, 1.00}). Which values were used for each benchmark, and how were they chosen? Was any tuning done on the evaluation sets?

  7. In Table 13, Query-Only gets 41.5% with Llama3.1-405B, which is below random. Do you know what causes this?

  8. How is the "Overall" score in Table 6 computed? It is lower than every individual dimension, which I didn't expect.

  9. For the gold rubric trees, how many annotators were involved, and did you measure agreement on the groupings?

  10. The distillation gain (69.43 vs. 69.14) and the RL gain (under 1 point, single run) are small. Could you run more seeds, or soften the claims about transfer to policy optimization?

7 comments
AuthorsOpenReview ↗

Response to Weakness 1A &Question 1

Thank you for recognizing the central motivation of our work: to understand which rubric design choices matter and why, beyond reporting end-to-end improvements. We appreciate your assessment of the controlled analyses and their practical value. Your comments also help identify where the evidence needs strengthening, and we address these concerns below with additional analyses and clarifications.

Weakness 1A & Question 1: There are no error bars, repeated runs, or significance tests anywhere, so I can't tell which of these are real.

Thank you for pointing out the lack of statistical evidence in the mechanistic analysis. We added three sets of supplementary results to address this concern.

First, we supplemented the main comparisons in Tables 1 and 2 with statistical tests and confidence intervals. We used query-level paired bootstrap confidence intervals and permutation tests, with Holm correction across the comparisons within each table. Bold values indicate positive differences; ★ denotes significance after Holm correction at p<0.05p<0.05.

Table 1. Coverage and Novelty comparisons

EvaluatorMetricComparisonΔ (pp)95% CIHolm-adjusted p
DeepSeekCoverageTREE − FLAT-MULTI+3.11[1.72, 4.51]0.00018 ★
DeepSeekCoverageTREE − FLAT+4.98[3.35, 6.66]0.00008 ★
DeepSeekNoveltyTREE − FLAT-MULTI+2.06[1.05, 3.09]0.00056 ★
DeepSeekNoveltyTREE − FLAT+1.97[0.68, 3.27]0.00630 ★
LlamaCoverageTREE − FLAT-MULTI+2.55[1.37, 3.76]0.00020 ★
LlamaCoverageTREE − FLAT+1.94[0.44, 3.42]0.01196 ★
LlamaNoveltyTREE − FLAT-MULTI+2.62[1.18, 4.07]0.00120 ★
LlamaNoveltyTREE − FLAT+4.79[2.92, 6.64]0.00008 ★

Table 2. Counterfactual evaluation comparisons

EvaluatorCounterfactual typeComparisonΔ (pp)95% CIHolm-adjusted p
DeepSeekSingle-leafPATH − FLAT+3.94[2.07, 5.81]0.00036 ★
DeepSeekSingle-leafPATH − TREE+4.21[2.27, 6.14]0.00036 ★
DeepSeekWithin-dimensionPATH − FLAT+0.54[−0.88, 1.96]0.51070
DeepSeekWithin-dimensionPATH − TREE+1.48[0.07, 2.97]0.16956
DeepSeekCross-dimensionPATH − FLAT+4.26[2.84, 5.75]0.00012 ★
DeepSeekCross-dimensionPATH − TREE+11.78[9.80, 13.77]0.00012 ★
LlamaSingle-leafPATH − FLAT+3.34[1.34, 5.28]0.00396 ★
LlamaSingle-leafPATH − TREE+5.01[2.94, 7.14]0.00020 ★
LlamaWithin-dimensionPATH − FLAT+2.29[1.08, 3.57]0.00185 ★
LlamaWithin-dimensionPATH − TREE+2.77[1.42, 4.12]0.00042 ★
LlamaCross-dimensionPATH − FLAT+1.35[0.00, 2.70]0.16956
LlamaCross-dimensionPATH − TREE+3.48[1.85, 5.11]0.00036 ★

Second, we performed three independent end-to-end runs of ARTS-Hybrid and Direct Judge on RB Chat to examine both their relative performance and variation across runs.

MethodRun 42 (%)Run 123 (%)Run 2026 (%)Mean (%) ± SD (pp)
Direct Judge86.0386.3186.3186.22 ± 0.16
ARTS-Hybrid90.5090.2288.2789.66 ± 1.22

Third, we repeated the shuffled-path control with two sampling/shuffling seeds:

EvaluatorCounterfactual typeSeed 1: CorrectSeed 1: ShuffledSeed 2: CorrectSeed 2: Shuffled
DeepSeek-V4-Flash-GASingle-leaf23.7723.63 (−0.13)21.2321.50 (+0.27)
DeepSeek-V4-Flash-GAWithin-dimension12.5511.61 (−0.94)12.2811.88 (−0.40)
DeepSeek-V4-Flash-GACross-dimension27.6625.67 (−1.99)27.0925.11 (−1.99)
Llama3.1-405B-InstructSingle-leaf35.9134.05 (−1.87)37.5236.98 (−0.53)
Llama3.1-405B-InstructWithin-dimension14.7113.09 (−1.62)15.6513.77 (−1.89)
Llama3.1-405B-InstructCross-dimension43.2642.41 (−0.85)44.2642.55 (−1.70)

All experimental results require us to reproduce and run the corresponding evaluation pipelines. Within the rebuttal period, we therefore prioritized statistical analyses of the key comparisons and the repeated runs reported above. Repeated experiments do not yet cover every setting, and we appreciate the reviewer’s understanding.

AuthorsOpenReview ↗

Response to Weakness 1B & Question 2

Weakness 1B & Question 2: All four settings in Table 3 land around 48–50% pairwise accuracy, which is close to chance, so it's hard to say that semantic filtering plus calibration "works best" when nothing really works on that set.

The near-50% accuracy in Table 3 should be considered in the context of RubricBench’s task design. The benchmark selects difficult response pairs involving complex requirements, misleading surface features, and reasoning errors. Its original Table 2 also reports several overall accuracies below 50%:

MethodBackboneRubricBench Overall Accuracy (%)
Vanilla JudgeGPT-4o-mini40.2
Vanilla JudgeDeepSeek-v3.238.8
TICKGPT-4o-mini45.2
OpenRubricGPT-4o-mini46.7

These results show that near- or below-chance performance is not unique to our experiments and also reflects the difficulty of RubricBench for existing evaluators. They provide context for the low absolute scores, rather than evidence of our method’s effectiveness.

Do the RQ2 conclusions still hold in a setting where the rubrics actually separate responses well?

We also examined semantic validation and calibration through end-to-end ablations on benchmarks where accuracy is substantially above 50%:

MethodRM-Bench ChatFocusJudgeBench
ARTS-Hybrid64.3485.5964.29
w/o Semantic50.3977.0461.71
w/o Calibration58.1486.1362.86

Removing semantic validation lowers accuracy on all three benchmarks. Removing calibration lowers accuracy on RM-Bench Chat and JudgeBench, but improves it slightly on Focus. These results provide complementary evidence beyond the near-chance RubricBench setting: semantic validation shows a consistent positive direction across these benchmarks, while the benefit of calibration depends on the evaluation setting.

AuthorsOpenReview ↗

Response to Weakness 2 & Question 3, 10

Weakness 2 & Question 3, Question 10: The comparison with baselines isn't on equal footing, and the downstream claims go beyond the results.

Thank you for the suggestion. To address the compute-budget concern, we added a Flat-Iterative baseline that matches the existing TREE runs’ total token budget and final criterion count. The budget includes all input and output tokens at every iteration. We also report actual token usage, criterion counts, and rubric lengths. The results are shown below:

MethodCoverage (%) ↑Novelty (%) ↑Mean input tokensMean output tokensMean criterion countMean rubric length (tokens)
TREE59.3410.577,364.021,653.049.72302.42
Flat-Iterative40.0010.716,240.672,394.449.72272.57

Across all samples, Flat-Iterative consumed 95.83% of TREE’s total construction tokens. On the 1,430 paired records where both methods completed successfully and satisfied the constraints, TREE achieved 59.34% Coverage, compared with 40.00% for Flat-Iterative—a gain of 19.34 percentage points (95% CI: [17.57, 21.13]).

These results show that TREE retains a clear coverage advantage with comparable actual token consumption and matched final criterion counts, even when the flat baseline can integrate and refine criteria across rounds, providing further support for hierarchical construction.

For related methods, we added comparisons with RRD and its variants. The table below reports JudgeBench accuracy using Llama3.1-405B as the evaluation judge. ARTS-Hybrid achieves 67.43%, exceeding the strongest RRD variant, RRD_WU (64.80%), by 2.63 percentage points.

MethodJudgeBench Accuracy (%)
Base Model57.40
LLM Rubrics (w/o resp.)41.50
LLM Rubrics (w. resp.)56.00
Chasing the Tail56.90
RRD_uniform61.10
RRD_LLM63.50
RRD_WU64.80
ARTS-Hybrid (ours)67.43

For RubricHub, its post-trained model results reflect both the constructed training data and updates to model parameters. Our evaluation instead examines how rubric construction affects evaluation performance with fixed models. These are different experimental settings, so its post-training scores cannot be directly included in the current accuracy comparison.

For the concurrent tree-based method, a reproducible implementation and complete evaluation configuration are not yet available, so we have not included it in the end-to-end comparison. We will expand the discussion in the revised manuscript and clarify the differences in research goals and method design.

Regarding distillation and RL, Tables 7 and 8 suggest that rubrics useful for evaluation do not necessarily provide effective supervision for distillation or reward signals for RL. Our experiments therefore examine not only improvements over the base model, but also the effects of different rubric sources as supervision. We will soften our claims about transfer to policy optimization and explicitly frame these experiments as an investigation of this suitability, rather than evidence of stable, general downstream improvements.

To examine whether the RL results depend on the original 7B evaluator, we re-evaluated the saved answers using Qwen2.5-32B-Instruct. The Thinking-mode HealthBench scores are shown below:

Reward sourceQwen2.5-7B-InstructQwen2.5-32B-Instruct
Base model38.1457.26
ARTS-Hybrid38.9357.32

We also report the significance test results in the table below.

ComparisonΔ (pp)Paired 95% CIUnadjusted pHolm-adjusted p
ARTS-Hybrid − Base model+0.79[+0.21, +1.37]0.008210.09852
ARTS-Hybrid − Direct-Prompt+0.84[+0.25, +1.43]0.004850.06305
ARTS-Hybrid − OpenRubrics Expert+0.59[−0.01, +1.20]0.055050.60554
ARTS-Hybrid − Q-World+1.28[+0.71, +1.87]0.000020.00028

Given the limited rebuttal period and the slow progress of the RL evaluations across multiple runs, we provide significance tests on the available results to assess the uncertainty of the observed differences. We will include results from multiple runs in the final version.

In addition, we conducted evaluations using two random seeds. The SFT results are reported below as mean ± standard deviation across the two runs.

SFT SupervisionMean (%) ± SD (pp)
Direct-Prompt Rubrics66.58 ± 0.40
OpenRubrics Expert Rubrics67.43 ± 1.22
Q-World Rubrics66.29 ± 2.02
ARTS-Hybrid Rubrics69.29 ± 0.21
Reviewer_z2peOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies rubric-based LLM evaluation through three design dimensions: criterion organization, criterion operationalization, and score aggregation. It proposes ARTS-Hybrid, which combines hierarchical rubric construction, semantic filtering, and response-based calibration using two judges and six synthetic responses. Experiments cover preference evaluation, intrinsic rubric quality, rubric-generator distillation, and rubric-based policy optimization.

Critical strengths
  1. Useful decomposition of the problem. Separating construction, criterion-level judgment, and aggregation is a productive way to study rubric effectiveness rather than treating rubric generation as a black box.

  2. Interesting hierarchy experiment. The controlled violation-detection experiments distinguish PATH conditioning from TREE routing. The observation that routing can suppress downstream detection is informative, and the shuffled-path control is a worthwhile addition.

  3. Broad empirical coverage. ARTS-Hybrid performs best on five of six settings in Table 5, and Table 13 provides additional cross-judge evidence. The paper also includes semantic-validation and calibration ablations in Table 14.

Critical weaknesses

1. The main structural control does not establish the claimed compute-independent benefit

FLAT-MULTI independently generates multiple flat rubrics and averages their performance. It does not combine, deduplicate, or refine their criteria. TREE, in contrast, incorporates information from multiple calls into one constructed rubric. Thus, matching call counts does not give the flat baseline an equivalent opportunity to use its computation.

A stronger control would iteratively expand a flat rubric, or merge multiple flat generations under the same final criterion and token budgets. Actual criterion counts should also be reported: matching a maximum number of leaves does not establish that coverage gains are independent of rubric length.

The observed coverage differences between TREE and FLAT-MULTI are only 0.61 and 2.77 percentage points for the two evaluators in Table 1, without uncertainty estimates. These results do not justify the strong attribution to hierarchical organization.

2. The operational definition of criterion effectiveness is insufficiently validated

Variance across response scores measures separation, not whether the separation is correct. A criterion that systematically favors the inferior response can have high variance. Likewise, agreement between judges does not establish validity: both judges may share the same bias.

This matters because the calibration responses differ simultaneously in model family and prompted quality level. Selection may reward sensitivity to stylistic or model-family artifacts rather than task quality. The independent quality checks in Figure 3 establish a group-level gradient, but do not show that the selected criteria distinguish substantive quality rather than these confounds.

3. The calibration objective has conceptual and scaling problems

The “judge stability” term in Equation 17 compares only the judges’ response-score variances. Two judges can rank responses in opposite orders while having identical variances and therefore receive maximal stability. This does not substantiate the claimed consistency of discrimination patterns.

The scales of the objective terms also deserve analysis. With scores in [0,1], response variance is at most 0.25. Under population variance, the variance across these judge-specific variances is at most 0.015625, making S(C)S(C) lie approximately between 0.9846 and 1. Its contribution therefore changes very little despite receiving the largest coefficient, 0.45. No term-level ablation or sensitivity analysis explains this design.

The redundancy penalty is also questionable. Distinct, necessary criteria can legitimately have correlated scores on six globally strong/medium/weak responses. Penalizing their maximum correlation risks suppressing valid criteria, and six observations provide a fragile basis for this statistic. Equation 20 should additionally specify distinct criterion pairs and handling of constant-score vectors.

4. Baselines and compute accounting are not strong enough

The end-to-end comparisons include simple rubric baselines and Q-World, but omit several directly relevant refinement approaches discussed in the paper, including RRD, RubricHub, and synthetic-evidence-based rubric evolution. Given the incremental ingredients, comparisons against closely related methods are important for establishing novelty.

ARTS-Hybrid also receives substantial additional inference resources: six response generations, two calibration judges, repeated candidate scoring, semantic checks, and global repairs. Using the same main generator and final judge does not make the comparison compute matched. There are no reported construction costs, token counts, latency measurements, or quality–cost curves. A budget-matched flat refinement baseline is particularly important.

5 Minor errors

Error: Equation (20) omits the restriction c≠c′c \ne c'. Including self-pairs makes the maximum positive Spearman correlation equal to 11 whenever a nonconstant criterion profile is present, so the formula cannot distinguish redundant from nonredundant multi-criterion decompositions. The maximization should be over distinct criteria. Location: Appendix B.1, “Score redundancy,” Equation (20), page 16; this penalty is used in Equation (3).

Questions for authors
  1. How are candidate child criteria assembled into alternative decompositions, and how many decompositions are evaluated?
  2. Which exact settings from the sets in Table 11 are used for each experiment, and on what development data are they selected?
  3. What are the benchmark sample sizes, response-order randomization procedures, and general tie-handling rules?
  4. What dataset and aggregation formula produce the intrinsic-quality “Overall” scores?
  5. Which dataset is evaluated in Table 13?
6 comments
AuthorsOpenReview ↗

Response to Weakness 1

Weakness 1: The main structural control does not establish the claimed compute-independent benefit

Thank you for highlighting the issues of statistical uncertainty and compute-budget control. We supplemented the comparisons in Table 1 with paired statistical analyses on the common samples with complete scores for both methods. We used query-level paired bootstrap confidence intervals and permutation tests, applying Holm correction across the eight comparisons. As shown below, all eight comparisons remain statistically significant after correction.

EvaluatorMetricComparisonΔ (pp)95% CIHolm-adjusted p
DeepSeekCoverageTREE − FLAT-MULTI+3.11[1.72, 4.51]0.00018
DeepSeekCoverageTREE − FLAT+4.98[3.35, 6.66]0.00008
DeepSeekNoveltyTREE − FLAT-MULTI+2.06[1.05, 3.09]0.00056
DeepSeekNoveltyTREE − FLAT+1.97[0.68, 3.27]0.00630
LlamaCoverageTREE − FLAT-MULTI+2.55[1.37, 3.76]0.00020
LlamaCoverageTREE − FLAT+1.94[0.44, 3.42]0.01196
LlamaNoveltyTREE − FLAT-MULTI+2.62[1.18, 4.07]0.00120
LlamaNoveltyTREE − FLAT+4.79[2.92, 6.64]0.00008

These results provide statistical support for TREE’s advantages in Coverage and Novelty on the paired samples above.

Regarding attribution, we agree that matching call counts alone does not separate the benefit of hierarchical organization from that of accumulating information across rounds. We therefore added Flat-Iterative: at each round, it reads the query and the current complete flat rubric, adds missing criteria, splits overly broad criteria, and merges or removes duplicates. This gives the flat baseline the opportunity to incorporate information from multiple rounds into its final rubric.

Because Flat-Iterative and TREE differ in the amount of information processed and the cost per call, we compare them under the same total token budget and final criterion-count constraint. The budget includes both input and output tokens, including the full rubric passed back in at each iteration. We also report actual input/output token usage, final criterion counts, and rubric lengths. The results and actual construction costs are shown below:

MethodCoverage (%) ↑Novelty (%) ↑Mean input tokensMean output tokensMean criterion countMean rubric length (tokens)
TREE59.3410.577,364.021,653.049.72302.42
Flat-Iterative40.0010.716,240.672,394.449.72272.57

Across all samples, Flat-Iterative consumed 95.83% of TREE’s total construction tokens. On the 1,430 paired records where both methods completed successfully and satisfied the constraints, TREE achieved 59.34% Coverage, compared with 40.00% for Flat-Iterative, a gain of 19.34 percentage points (95% CI: [17.57, 21.13]).

These results show that TREE retains a clear coverage advantage with comparable actual token consumption and matched final criterion counts, even when the flat baseline can integrate and refine criteria across rounds, providing further support for hierarchical construction.

AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2: The operational definition of criterion effectiveness is insufficiently validated

We agree with the reviewer’s observation, which is also consistent with the issue illustrated in Table 3: higher score variance does not imply that responses are distinguished in the correct direction, and agreement between judges does not guarantee valid judgments. We therefore do not treat variance or agreement as sufficient conditions for criterion validity. Instead, we use these signals to help select candidates subject to semantic constraints.

Table 3 shows the limitations of relying on behavioral calibration alone. Calibration achieves the highest discrimination and lowest cross-judge variance, yet 17.84% of its selected revisions violate the semantic validity constraints. Optimizing scoring behavior can therefore select criteria that depart from the task requirements. This motivates ARTS-Hybrid’s approach of semantic filtering followed by behavioral calibration. Semantic + Calibration reduces the proportion of selections violating these constraints to zero while achieving the highest average preference accuracy.

MethodDisc. ↑Judge Var. ↓Invalid (%) ↓Llama Accuracy (%)Qwen Accuracy (%)Avg. Accuracy (%) ↑
Direct0.51050.2749—48.2747.8748.07
Semantic0.52070.27260.0049.8747.0048.44
Calibration0.54500.251017.8448.2749.7349.00
Semantic + Calibration0.52950.26260.0049.3350.5349.93

To examine the coupling between generator family and target quality tier, we added three experiments covering within-model quality differences, replacement of the Strong-tier generator, and changes to the calibration panel’s size and composition.

First, we used Qwen3-30B-A3B-Instruct-2507, Llama-3.1-70B-Instruct, and Mistral-7B-Instruct-v0.2 separately to generate six calibration responses per query: two responses for each target tier—Strong, Medium, and Weak. We then evaluated their quality using DeepSeek-V4-Flash-GA and Gemini-3.8-flash. The table below reports the equally weighted average of the two evaluators’ scores.

Calibration panelStrongMediumWeakStrong − Weak [95% CI]
Original construction63.9765.9439.1224.85 [22.97, 26.75]
Qwen3-30B-A3B-Instruct-2507 only63.9767.4456.617.36 [6.20, 8.58]
Llama-3.1-70B-Instruct only65.9265.9457.278.65 [7.59, 9.73]
Mistral-7B-Instruct-v0.2 only47.7248.1839.128.60 [7.55, 9.66]

With the generator held fixed, the Strong–Weak gap remains positive for all three models, with confidence intervals excluding zero. Thus, the observed quality differences do not depend entirely on changes in model family. The original construction produces a substantially wider Strong–Weak gap of 24.85 points, supporting our motivation to combine different generators and quality-control prompts to cover a broader range of response quality.

Second, we replaced Qwen3-30B-A3B-Instruct-2507 with GPT-4o-mini as the Strong-tier generator, regenerated the calibration responses, and ran ARTS-Hybrid. With Llama-3.1-405B-Instruct as the evaluation judge, the results are:

MethodJudgeBench Accuracy (%)
ARTS-Hybrid (ours)69.71
ARTS-Hybrid (GPT-4o-mini for Strong responses)67.43

Finally, we removed the Medium tier, retaining only two Strong responses from Qwen3-30B-A3B-Instruct-2507 and two Weak responses from Mistral-7B-Instruct-v0.2. We reran the complete ARTS-Hybrid pipeline with GLM-4.7-Flash, keeping the remaining settings fixed.

BenchmarkARTS-Hybrid: Six responses (%)Four responses (%)
JudgeBench64.2961.71

These supplementary experiments support our choice of the six-response calibration panel.

AuthorsOpenReview ↗

Response to Weakness 3-5

Weakness 3: The calibration objective has conceptual and scaling problems

Thank you for pointing out the issue with the scales of the objective terms. Under the population-variance definition, S(C)S(C) does have a narrow range. In ARTS-Hybrid, however, we first check whether candidate decompositions satisfy the semantic constraints and then compare feasible candidates using the objective score. Thus, SS serves as an auxiliary selection signal. Its actual influence depends on the weighted score differences between candidates, rather than the coefficient alone.

The same distinction applies to RR. We agree that correlated scores do not imply semantic redundancy: distinct, necessary criteria may vary together with overall response quality. We do not treat RR as sufficient evidence of redundancy. It is a soft penalty among semantically feasible candidates, not a rule that directly removes correlated criteria. Candidate sets must still satisfy the semantic coverage requirements.

We also performed ablations that remove each term and rerun the complete search. The results for construction are shown below:

SettingJudgeBench Accuracy (%)
Full64.29
without_S61.43
without_R62.86

In this run, removing SS and RR reduced accuracy by 2.86 and 1.43 percentage points, respectively.

The purpose of SS and RR is to supplement semantic constraints with behavioral information when comparing candidate decompositions, rather than independently determine criterion validity. We will clarify this role in the revised manuscript and distinguish the objective coefficients from the actual variation in their weighted terms.

Weakness 4: Baselines and compute accounting are not strong enough

Thank you for the suggestion. We added comparisons with related methods and further examined the benefits of structural design under controlled construction budgets.

First, we added RRD and its weighting variants. The table below reports JudgeBench accuracy using Llama3.1-405B as the evaluation judge. ARTS-Hybrid achieves 67.43%, exceeding the strongest RRD variant, RRD_WU (64.80%), by 2.63 percentage points.

MethodJudgeBench Accuracy (%)
Base Model57.40
LLM Rubrics (w/o resp.)41.50
LLM Rubrics (w. resp.)56.00
Chasing the Tail56.90
RRD_uniform61.10
RRD_LLM63.50
RRD_WU64.80
ARTS-Hybrid (ours)67.43

For RubricHub, its post-trained model results reflect both the constructed training data and updates to model parameters. Our evaluation instead examines how rubric construction affects evaluation performance with fixed models. These are different experimental settings, so its post-training scores cannot be directly included in the current accuracy comparison.

Synthetic-evidence-based rubric evolution iteratively refines rubrics using synthetic evidence, whereas our method searches over candidate decompositions subject to semantic constraints and uses calibration responses to guide selection. Although both involve rubric refinement, they differ in feedback, optimization procedures, and inference budgets. Their reported results therefore do not constitute a comparison under matched conditions. We will clarify these distinctions in the related work.

We also agree that using the same generator and final judge does not ensure matched computation. We therefore added Flat-Iterative, evaluated under the same total token budget and final criterion-count constraint. The results favor tree-based construction even under these matched constraints. Please see our response to Weakness 1 for the experimental details and results.

Weakness 5: Minor Errors

We have corrected Equation (20) in the revised manuscript to restrict the maximization to distinct criterion pairs, c≠c′c \ne c'.

Comments on the paper

2 comments

Thank you for your follow-up and acknowledging that our responses have addressed some of your concerns. Following your suggestions, we have incorporated the additional experiments and clarifications into the revised manuscript, with the supplementary experiments presented in the Appendix. We would appreciate it if you could let us know which concerns remain unresolved, and we would be happy to address them during the discussion phase. Thank you again for your time and consideration.

AuthorsOpenReview ↗

Rebuttal Summary

We thank all four reviewers for their constructive feedback. Their concerns largely centered on the statistical reliability of the results, the fairness of the structural comparisons, the design of response-based calibration, and the clarity of the evaluation protocol. We have addressed these shared concerns through additional experiments, statistical analyses, and methodological clarifications, which have been incorporated into the revised manuscript. Supplementary experiments are presented in the appendix.

  • Statistical reliability. We added query-level paired confidence intervals and significance tests for key comparisons, together with evaluations using additional judge models.
  • Fairness of structural comparisons. We introduced Flat-Iterative, a baseline that iteratively refines a flat rubric. With comparable construction token usage and matched final criterion counts, tree-structured rubrics maintained a clear advantage in coverage.
  • Calibration design and criterion validity. We conducted sensitivity analyses varying the calibration response generator and the size and composition of the calibration panel, alongside ablations of the calibration objective. We also clarified the limitations of score variance as a measure of ranking correctness and the role of semantic validation in screening candidate criteria.
  • Evaluation protocol and reproducibility. We clarified the role of RubricBench in the mechanism analysis and provided further details on hyperparameters, sample sizes, and tie-handling procedures.

The follow-up discussion also reflected recognition of these contributions and responses. Reviewer 1kHW emphasized the value of our controlled analyses for understanding why rubrics work and increased their score after reviewing the rebuttal. Reviewer NFr4 acknowledged that our responses had addressed some of their concerns and requested that the relevant experiments and explanations be included in the revision. These additions are now incorporated into the revised manuscript.

We appreciate the reviewers’ time and constructive engagement, and we thank the Area Chair for considering our responses and the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.