- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether apparent oracle improvements across related language-model policies reflect reproducible specialization or finite-response sampling noise. It introduces a within-prompt re-deal test and a margin-preserving conditional test under an additive-logit model. Simple coverage identities distinguish an oracle opportunity from gains achievable by a mixture or router. Experiments examine seed variation, cross-dataset training, stronger training, multiple response budgets, and several deployment rules, with simulations and positive controls characterizing the tests.
Critical strengths
-
The evaluation perspective is distinctive and valuable. Separating a noisy estimated oracle gap, a population specialization gap, and a deployable routing gain clarifies what policy-pool experiments can actually establish.
-
The paper supports that perspective with a careful statistical investigation: independent selection/scoring, power simulations, misspecification checks, larger response budgets, and positive controls. These analyses provide substantial insight into when apparent specialization can be misleading. The explicit distinctions among the statistical quantities and the reported limitations help bound the interpretation of the results.
Critical weaknesses
-
The paper could make its operational guidance more accessible. At k=16, Table 2 detects a population gap near 0.05 in only about half of the simulated alternatives; the paper also documents link-function sensitivity and limits of subtracting the null mean. The paper already provides useful analyses of these limits; a concise explanation of what a non-significant result permits practitioners to conclude would make the contribution easier to apply. Uncertainty in the estimated break-even routing requirement L/D would also help distinguish a statistical diagnostic from a practical routing decision.
-
The light-versus-strong training comparison changes both the training configuration and the training-time verifier. Appendix G states that the primary light Geometry3K models used a checker with a degree-symbol issue, whereas stronger training used the corrected checker. Correct re-scoring of evaluations does not remove this training-time difference. A matched corrected-checker control would be informative; alternatively, explicitly describing the comparison as two recipes with different verifiers would appropriately limit its interpretation.
Questions for authors
-
How should a practitioner interpret a non-significant result at k=16 given the measured power, and how does that guidance change at k=64?
-
How uncertain is the estimated break-even requirement L/D, and does accounting for that uncertainty change the practical routing conclusions?
-
Could you clarify whether a corrected-verifier light-training control is available? If not, explicitly noting both differences between the recipes would address the attribution concern.