- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Judge CIA, which selects a jury from a heterogeneous pool of LoRA adapters sharing the same backbone. Adapters are ranked by validation accuracy and filtered based on pairwise error correlation before equal-weight voting. The paper analyzes how correlation affects an upper bound on majority-vote error and evaluates the method across three RAGBench tasks and four backbones, with additional analyses of correlation thresholds, jury size, ranking stability, and error-weighted selection.
Critical strengths
The paper addresses a practical question about balancing individual competence and shared errors in small-model judge ensembles. The experiments include matched-size comparisons and sensitivity analyses while distinguishing overall accuracy gains from class-specific trade-offs.
Critical weaknesses
-
The benefit of correlation-aware selection is not sufficiently established. The main comparison across 12 settings is against Top-1, which does not isolate the contribution of correlation filtering. Matched-size comparisons against accuracy-only selection cover only four cases, and all four confidence intervals include zero. Baselines that directly select members to optimize validation ensemble accuracy are also absent. The evidence therefore does not yet establish that correlation filtering is a more effective jury-selection strategy.
-
The practical benefit of heterogeneous pool construction is not directly demonstrated. The construction study reports lower error correlation in only one backbone-task setting, without systematically comparing final jury accuracy between seed-only and heterogeneous pools under the same selection rule.
Questions for authors
-
Please report matched-size comparisons of CIA, accuracy-only top-k, and random selection across all 12 settings, including per-setting results and a confidence interval for the average difference.
-
Please add a greedy selection baseline that directly optimizes validation voting accuracy, using the same candidate pool and jury-size budget.
-
Please compare seed-only and heterogeneous pools under matched training budgets across multiple backbone-task settings, applying the same selection rule and reporting final jury accuracy.