- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper conducts experiments across multiple preference evaluation benchmarks and examines the intrinsic quality of the generated rubrics and their applications in model distillation and reinforcement learning. ARTS-Hybrid achieves the highest accuracy on five of the six main evaluation settings, while the cross-judge experiments and ablation studies also demonstrate promising results. The paper presents a clear research direction and a broad experimental evaluation, although some aspects regarding the independent contributions of its components, the validity of the calibration metrics, and the reliability of certain experimental improvements remain open to discussion.
Critical strengths
-
Relevant research problem. Rubric design directly affects the reliability of large language model evaluation. The paper investigates criterion organization, criterion-level effectiveness, and score aggregation, providing a systematic perspective on how different design choices influence evaluation performance.
-
Methodological novelty. ARTS-Hybrid combines adaptive tree search with semantic-first, response-based calibration, considering both the semantic validity and empirical scoring behavior of candidate criteria during hierarchical rubric construction. The integration of structured generation and behavioral calibration is well motivated.
-
Comprehensive experimental evaluation. The paper includes structural analyses, counterfactual violation detection, evaluation across multiple benchmarks, cross-judge experiments, and component ablations. It also investigates the use of generated rubrics in model distillation and reinforcement learning, providing empirical evidence from several perspectives.
Critical weaknesses
- The independent methodological novelty and contributions of key components are not sufficiently clear.
ARTS-Hybrid integrates hierarchical decomposition, semantic filtering, response-based calibration, and global refinement into a unified framework. However, hierarchical rubric construction and iterative refinement have already been explored in related works such as Q-World and RRD. The main methodological contribution appears to lie in combining adaptive tree search with semantic-first calibration, but the individual contributions of these components are not sufficiently disentangled.
Table 14 primarily examines the effects of semantic validation and response-based calibration, without separately analyzing adaptive node expansion, priority scheduling, or global refinement. In Table 1, Tree improves coverage over the call-matched Flat-Multi baseline by only 0.61 and 2.77 percentage points under the two judge models, respectively. Although these results indicate some advantages of hierarchical generation, it remains unclear to what extent the improvements arise from the tree structure itself rather than repeated candidate generation and exploration. The current evidence does not fully clarify the necessity of the individual design choices or their incremental value over existing approaches.
- The relationship between the calibration metrics and actual evaluation effectiveness is not sufficiently validated.
The paper measures discriminative power using score variance across different responses. However, higher variance does not necessarily indicate that the scores are aligned with actual response quality. For example, a criterion may assign lower scores to high-quality responses and higher scores to low-quality responses while still exhibiting high discriminative power. Similarly, the stability metric in Appendix Equations (16)–(17) compares score variances across judge models, but similar variances do not imply consistent response rankings. Two judges may produce opposite rankings while still receiving a high stability score.
These metrics directly influence candidate criterion selection in ARTS-Hybrid, making their validity closely related to the core methodology. In Table 3, Semantic + Calibration achieves an average pairwise accuracy of 49.93%, an improvement of 1.86 percentage points over Direct at 48.07%, but the overall accuracy remains close to 50%. The current results do not sufficiently establish whether the proposed discrimination and stability metrics reliably reflect the quality of actual preference judgments.
- The statistical reliability of some experimental improvements and their downstream effectiveness remain uncertain.
Table 5 shows that ARTS-Hybrid achieves the highest accuracy on five of the six main evaluation benchmarks. However, the improvements over the strongest competing baseline on each benchmark range from 0.58 to 2.41 percentage points, and the method does not achieve the best result on PreciseIF. Although the cross-judge experiments in the appendix show more substantial gains, the smaller differences in the main experiments are not accompanied by sufficient statistical uncertainty analysis, leaving their reliability unclear.
The downstream improvements are also relatively limited. In Table 7, supervised fine-tuning with ARTS-Hybrid rubrics increases JudgeBench accuracy from 69.14% to 69.43%, a gain of only 0.29 percentage points. In Table 8, the reinforcement learning experiments improve performance over the base model by 0.79 and 0.32 points under the Thinking and Non-Thinking settings, respectively. These results suggest potential benefits of using the generated rubrics for model optimization, but the current evidence does not sufficiently establish the statistical reliability or practical significance of these relatively small improvements.
Questions for authors
-
What are the main methodological contributions of ARTS-Hybrid compared with existing hierarchical rubric generation methods such as Q-World and RRD? What are the respective roles of adaptive tree search, node priority scheduling, and global refinement in the complete framework?
-
Why does the paper use score variance to measure discriminative power rather than directly considering ranking consistency with response quality levels? How does the current stability metric account for cases where two judge models have identical score variances but produce opposite rankings?
-
The paper constructs six calibration responses for each query, consisting of two strong, two medium, and two weak responses. What motivates this sample size and quality distribution? How sensitive are the calibration results to changes in the number and composition of calibration responses?
-
Is statistical uncertainty analysis available for the relatively small improvements reported in Tables 5, 7, and 8? To what extent do these results support consistent improvements in evaluation performance and downstream model optimization?
6 comments
Response to Weakness 3
Weakness 3: The statistical reliability of some experimental improvements and their downstream effectiveness remain uncertain.
For the main evaluation, we performed 10,000 query-level paired bootstrap resamples, sampling all response pairs belonging to the same query together and using identical sampling indices for both methods. Compared with Q-World, ARTS-Hybrid improves accuracy by 8.10 percentage points on RB Chat (95% CI: [4.31, 12.02]) and 2.56 percentage points on Focus (95% CI: [0.40, 4.71]).
For distillation, Table 7 examines not only the evaluation performance of the rubrics themselves, but also whether they can serve as supervision for training a separate rubric generator. Under the same training data and settings, distillation using other rubric sources yields accuracies below the base generator’s 69.14%, while ARTS-Hybrid reaches 69.43%. This comparison suggests that the source of rubrics used for distillation can affect training outcomes, and generated rubrics do not automatically provide effective supervision. ARTS-Hybrid achieves the highest accuracy in this experiment while maintaining the base generator’s performance level.
To further assess the stability of the results, we report the mean accuracy and standard deviation across two runs, as shown below.
| SFT Supervision | Mean (%) ± SD (pp) |
|---|---|
| Direct-Prompt Rubrics | 66.58 ± 0.40 |
| OpenRubrics Expert Rubrics | 67.43 ± 1.22 |
| Q-World Rubrics | 66.29 ± 2.02 |
| ARTS-Hybrid Rubrics | 69.29 ± 0.21 |
Similarly, the RL experiment examines whether rubrics from different sources are suitable reward signals for post-training. Table 8 shows that different rubric sources do not produce consistent training gains: Direct-Prompt performs slightly below the base model in both settings, while the effects of Q-World and expert rubrics vary across settings. ARTS-Hybrid achieves the highest score in both settings, exceeding the base model.
We provide two additional analyses. First, we conducted paired significance tests for the Thinking setting using the original Qwen2.5-7B-Instruct evaluator. The improvements over Base, Direct-Prompt, and Q-World are significant in the unadjusted tests; the improvement over Q-World remains significant after Holm correction.
| Comparison | Δ (pp) | Paired 95% CI | Unadjusted p | Holm-adjusted p |
|---|---|---|---|---|
| ARTS-Hybrid − Base model | +0.79 | [+0.21, +1.37] | 0.00821 | 0.09852 |
| ARTS-Hybrid − Direct-Prompt | +0.84 | [+0.25, +1.43] | 0.00485 | 0.06305 |
| ARTS-Hybrid − OpenRubrics Expert | +0.59 | [−0.01, +1.20] | 0.05505 | 0.60554 |
| ARTS-Hybrid − Q-World | +1.28 | [+0.71, +1.87] | 0.00002 | 0.00028 |
Second, we re-evaluated the saved responses using the larger Qwen2.5-32B-Instruct evaluator. The Thinking-setting results are shown below:
| Reward source | Qwen2.5-7B-Instruct | Qwen2.5-32B-Instruct |
|---|---|---|
| Base model | 38.14 | 57.26 |
| ARTS-Hybrid | 38.93 | 57.32 |
With Qwen2.5-32B-Instruct, ARTS-Hybrid retains a positive point estimate relative to the base model.
Response to Question 1
Question 1: What are the main methodological contributions of ARTS-Hybrid compared with existing hierarchical rubric generation methods such as Q-World and RRD? What are the respective roles of adaptive tree search, node priority scheduling, and global refinement in the complete framework?
The methodological contribution of ARTS-Hybrid lies in integrating semantic constraints and response-based behavioral calibration into the step-by-step construction of rubrics, rather than introducing hierarchical decomposition or iterative revision for the first time. Building on the structured construction and refinement explored by Q-World and RRD, we emphasize that candidate decompositions should be semantically sound and exhibit useful scoring behavior on actual responses. We therefore first construct candidate sets that satisfy semantic constraints, then use the calibration objective to select a decomposition, rather than relying solely on the semantic quality of the generated text. RQ1–RQ3 examine the roles of structure, criterion selection, and aggregation, respectively, providing the basis for this design.
The three components serve distinct roles within the framework:
| Component | Role |
|---|---|
| Adaptive tree search | Determines which criteria require further decomposition, which set of child criteria to adopt, and when to stop. Atomic nodes that can be assessed independently remain leaves; other nodes are expanded within the depth and leaf budgets. |
| Node priority scheduling | Determines which node to expand next under a limited budget. It combines task importance and depth through to prioritize exploration; it does not determine which candidate decomposition is better. |
| Global refinement | Reviews the complete criterion set after local construction to address missing requirements, cross-branch duplication, and non-atomic criteria. Passing local checks does not guarantee that the final set is globally complete and non-redundant. |
Question 2: Why does the paper use score variance to measure discriminative power rather than directly considering ranking consistency with response quality levels? How does the current stability metric account for cases where two judge models have identical score variances but produce opposite rankings?
We use score variance to measure separation between responses, rather than explicitly requiring scores to follow the intended Strong–Medium–Weak ordering, for two reasons. First, these quality levels are controlled through generation prompts and do not constitute a verified ranking of individual responses. A response with higher overall quality may not perform better on every specific criterion. Second, ranking metrics do not capture score magnitudes: two judges may produce the same ranking while assigning different sizes to the gaps between responses. We therefore use variance as a proxy signal for behavioral calibration to help select candidates after semantic filtering, rather than as a guarantee that responses are distinguished in the correct direction.
Regarding identical variances with opposite rankings, we agree with the reviewer: the current variance-based stability metric cannot distinguish this case. More broadly, higher score variance does not imply correct discrimination, and agreement between judges does not guarantee valid judgments. We therefore do not treat variance or agreement as sufficient conditions for criterion validity, but use these signals to assist candidate selection under semantic constraints.
Table 3 illustrates a related limitation of relying on behavioral calibration alone. Calibration achieves the best discrimination and cross-judge variance metrics, yet 17.84% of its selected revisions violate the semantic validity constraints. This shows that optimizing scoring behavior can select criteria that deviate from task requirements, motivating ARTS-Hybrid’s approach of semantic filtering followed by behavioral calibration. Semantic + Calibration reduces this invalid-selection rate to zero while achieving the highest average pairwise accuracy.
| Method | Disc. ↑ | Judge Var. ↓ | Invalid (%) ↓ | Llama Accuracy (%) | Qwen Accuracy (%) | Avg. Accuracy (%) ↑ |
|---|---|---|---|---|---|---|
| Direct | 0.5105 | 0.2749 | — | 48.27 | 47.87 | 48.07 |
| Semantic | 0.5207 | 0.2726 | 0.00 | 49.87 | 47.00 | 48.44 |
| Calibration | 0.5450 | 0.2510 | 17.84 | 48.27 | 49.73 | 49.00 |
| Semantic + Calibration | 0.5295 | 0.2626 | 0.00 | 49.33 | 50.53 | 49.93 |
Response to Weakness 1& 2
Thank you for the suggestion. We will clarify that the paper’s main contribution is its analysis of rubric design mechanisms and the construction method derived from these findings. We do not claim that hierarchical decomposition or iterative refinement is itself new. We separately examine how criteria are organized, selected, and aggregated to explain which design choices help and under what conditions, rather than simply proposing a pipeline with higher end-to-end scores. This is also consistent with Reviewer 1kHW’s assessment of our contribution: “It tries to explain why rubrics work, not just show another method that scores higher.”
The methodological contribution of ARTS-Hybrid lies in translating these findings into adaptive search and behavioral calibration under semantic constraints. These components serve distinct roles; criteria are not selected solely for their ability to separate response scores.
To examine whether the benefits of tree construction arise from repeated generation and exploration, we added a Flat-Iterative control. Flat-Iterative reads the complete current criterion set at each round and adds missing criteria, splits overly broad criteria, and merges or removes duplicates. For each query, we match TREE’s total input/output token budget and final criterion count, including the cost of feeding the complete rubric back into the model at every iteration. We also report actual token usage and final rubric length. The results and actual construction costs are shown below:
Across all samples, Flat-Iterative consumed 95.83% of TREE’s total construction tokens. On the 1,430 paired records where both methods completed successfully and satisfied the constraints, TREE achieved 59.34% Coverage, compared with 40.00% for Flat-Iterative.
These results show that TREE retains a clear coverage advantage with comparable actual token consumption and matched final criterion counts, even when the flat baseline can integrate and refine criteria across rounds, providing further support for hierarchical construction.
We agree with the distinction raised by the reviewer, which is also reflected in Table 3: higher score variance does not imply that responses are distinguished in the correct direction, and similar variances across judges do not imply consistent response rankings. We therefore do not treat these metrics as sufficient conditions for criterion validity. Instead, we use them to help compare candidate decompositions under semantic constraints.
Table 3 illustrates the limitations of relying on behavioral calibration alone. Calibration achieves the best discrimination and cross-judge variance metrics, yet 17.84% of its selected revisions violate the semantic validity constraints. This shows that optimizing scoring behavior can select criteria that deviate from task requirements, motivating our approach of semantic filtering followed by behavioral calibration. We therefore also evaluate preference judgments on independent response pairs.
The near-50% accuracy in Table 3 should also be understood in the context of RubricBench’s design. The dataset specifically selects difficult response pairs involving complex requirements, misleading surface features, and reasoning errors. Table 2 of the original RubricBench paper also reports several overall accuracies below 50%:
These results show that performance near or below chance is not unique to our setting and reflects the challenges this benchmark poses for existing evaluation methods.
RQ2 therefore examines the complementary roles of semantic filtering and behavioral calibration, it does not claim to have solved reliable preference judgment on RubricBench. We will clarify this scope in the revised manuscript.