acceptodds
Withdrawn from ICLR 2027

Correlation-Aware Ranking And Filtering For Heterogeneous Small-Model Judge Ensembles

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Shared errors limit the benefits of combining small language model judges. Judge CIA (Correlation-Informed Agents) constructs heterogeneous same-backbone LoRA pools, ranks adapters by validation accuracy, and filters pairwise error correlation before majority voting. Our analysis separates individual competence from effective jury size in a majority-error upper bound, identifying conditions under which a smaller, less correlated jury has a tighter bound. Across 12 RAGBench backbone–task settings, CIA improves accuracy over the validation-selected Top-1 adapter by 1.99 percentage points on average (paired-bootstrap 95% interval , conditional on fixed pools and selections), with gains up to 7.00 points. In four illustrative matched-size comparisons, CIA shows point-estimate test-accuracy gains of 2.81–4.74 points over accuracy-only selection, accompanied by lower test error correlation. For high-stakes judging, a slice-risk decomposition motivates weighted ranking. In a constructed compliance study, weighting reduces mean critical-example error from 2.26% to 1.65% and overall error from 2.60% to 2.48%. Public class weighting reduces target-class error by 8.17 points, with increased non-target error.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, Small Language Models, Ensemble, Correlation-Aware Selection, Error Correlation, LoRA Adapters, Majority Voting, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DUgVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Judge CIA, which selects a jury from a heterogeneous pool of LoRA adapters sharing the same backbone. Adapters are ranked by validation accuracy and filtered based on pairwise error correlation before equal-weight voting. The paper analyzes how correlation affects an upper bound on majority-vote error and evaluates the method across three RAGBench tasks and four backbones, with additional analyses of correlation thresholds, jury size, ranking stability, and error-weighted selection.

Critical strengths

The paper addresses a practical question about balancing individual competence and shared errors in small-model judge ensembles. The experiments include matched-size comparisons and sensitivity analyses while distinguishing overall accuracy gains from class-specific trade-offs.

Critical weaknesses
  1. The benefit of correlation-aware selection is not sufficiently established. The main comparison across 12 settings is against Top-1, which does not isolate the contribution of correlation filtering. Matched-size comparisons against accuracy-only selection cover only four cases, and all four confidence intervals include zero. Baselines that directly select members to optimize validation ensemble accuracy are also absent. The evidence therefore does not yet establish that correlation filtering is a more effective jury-selection strategy.

  2. The practical benefit of heterogeneous pool construction is not directly demonstrated. The construction study reports lower error correlation in only one backbone-task setting, without systematically comparing final jury accuracy between seed-only and heterogeneous pools under the same selection rule.

Questions for authors
  1. Please report matched-size comparisons of CIA, accuracy-only top-k, and random selection across all 12 settings, including per-setting results and a confidence interval for the average difference.

  2. Please add a greedy selection baseline that directly optimizes validation voting accuracy, using the same candidate pool and jury-size budget.

  3. Please compare seed-only and heterogeneous pools under matched training budgets across multiple backbone-task settings, applying the same selection rule and reporting final jury accuracy.

Reviewer_qshROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present a way to choose a group of small AI models to judge other models' answers. They aim to select judges that are accurate but make different mistakes. However, the experiments do not clearly show that this way of choosing the group is better than simpler alternatives.

Critical strengths

The selection procedure is easy to follow, and the authors test 12 combinations of models and tasks (§§3.1, 5.1). They also report cases where using a group makes the results worse and acknowledge several limits of the study.

Critical weaknesses
  • The main comparison does not directly test the paper's central idea. Table 1 compares a group of judges against a single judge. This can show that combining judges helps, but it does not establish that the proposed way of selecting them is useful. The more relevant comparison against simply choosing the most accurate judges appears in only four “illustrative” cases (§5.4, Table 2). The main text does not explain why these cases were chosen.

  • The authors acknowledge that "Direct ensemble-accuracy optimizers are not compared" (§6). These are methods that directly look for a group whose combined answers are accurate. This seems like an important missing comparison: without it, I cannot tell whether measuring shared mistakes offers an advantage over directly testing which group works best.

  • The mathematical discussion explains why judges making the same mistakes can limit the benefit of voting (§4). However, it does not establish that the proposed selection procedure works well. The main idea therefore depends heavily on the experiments, which are not comprehensive enough to support a strong contribution. The authors report 252 short fine-tuning runs for each model–task combination, with each run making one pass through at most 1,000 examples (§§3.1, 5.1). This involves real training work and produces many candidate judges. However, the number of candidates does not establish that the proposed selection rule is useful. More complete comparisons against simpler ways of choosing the judges are needed to show what this training effort is actually useful.

Questions for authors
  1. Across all 12 settings, how does the proposed method compare with simpler ways of choosing groups of the same size, including directly choosing a group for its combined accuracy?
  2. How were the four cases in Table 2 chosen, and were those choices made before examining their test results?
Reviewer_hM1QOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Judge CIA, a framework that constructs heterogeneous same-backbone LoRA judge pools, ranks adapters by validation accuracy, and filters highly correlated error patterns before equal-weight majority voting. Its theoretical analysis separates individual competence from an effective jury size that accounts for error correlation, illustrating when a smaller, less correlated ensemble can have a tighter majority-error bound. Across 12 RAGBench backbone selected best individual adapter by an average of 1.99 percentage points, while four illustrative matched-size comparisons favor correlation-aware selection over accuracy-only selection. Exploratory weighted-ranking experiments further demonstrate how member selection can prioritize specific error categories, although target-class improvements may come at the expense of non-target performance and do not establish reduced real-world harm.

Critical strengths

The paper addresses a clear and practically relevant problem: individually accurate judges may share errors, so effective ensemble selection should consider both individual competence and error complementarity.

CIA is simple, modular, and interpretable, combining heterogeneous LoRA pool construction, validation-based ranking, pairwise error-correlation filtering, and equal-weight majority voting.

Critical weaknesses

Evidence for the core selection mechanism remains limited. Matched-size comparisons cover only four settings, and all four confidence intervals include zero. Moreover, the uncertainty analysis conditions on fixed trained pools and selections, leaving the stability of the full pipeline unverified.

The high-stakes experiments do not fully validate the complete CIA framework or demonstrate reduced real-world harm. The constructed compliance study omits correlation filtering and involves few errors, while public class weighting improves target-class performance at the expense of non-target performance.

Questions for authors

Compared to greedy member selection that directly optimizes validation-set voting accuracy, how do CIA's accuracy, stability, and selection cost perform?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.