- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
RuDE predicts post-training performance through rubric-based response discrimination. Across 16 base/instruct model pairs and four domains, Pearson correlations range from 0.62 to 0.91. A three-model HealthBench RL experiment provides additional ranking evidence.
Critical strengths
-
The model-selection problem is useful, and the 28,683-example construction pipeline is substantial.
-
Cross-domain correlations, Spearman analysis, score confidence intervals, and controlled RL provide complementary evidence.
-
Construction diagnostics include failure rates and a 100-example human audit.
Critical weaknesses
-
Equation (2) specifies response likelihoods, while the implementation describes two-shot A/B selection. Table 11 shows an 8.6-point reversal effect for Qwen3-14B; the small signed average does not establish ranking robustness.
-
Released instruct models use heterogeneous training pipelines. Their correlations support retrospective prediction, but not algorithm-independent trainability. Controlled RL covers only three Qwen models and lacks sufficient recipe, train/test-split, and seed details.
-
Only one domain exceeds 0.90 Pearson correlation. HealthBench reports p < 0.001 in Figure 3/Table 15 but p = 0.005 in Figure 10. Correlation uncertainty and actual model-selection errors would be more informative than significance alone.
-
PPL reaches 0.93 on AdvancedIF using a smaller model subset. Compare predictors on identical models. Construction ablations should compare post-training predictive accuracy, not merely discrimination difficulty.
Questions for authors
-
What exact scoring implementation produced the results, and are rankings stable after averaging option orders?
-
What are the RL recipe, disjoint training/test sets, and seed uncertainty? Disclose RuDE/evaluation prompt overlap; task-specific evaluation does not itself require separate prompts.
-
Can you reconcile the statistics and compare predictors on identical model sets, including held-out families?