- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper benchmarks 13 prompting strategies for grading mathematical proofs across four open-weight LLM judges on two new expert-graded datasets (2,080 candidate solutions total, evaluated over 5 runs each). Because the best prompting strategy shifts depending on the judge model and dataset, the authors propose Mixture of Graders (MoG): a ModernBERT router that takes the problem and reference solution and outputs weights to combine all 13 protocol scores. They report that MoG beats the best single protocol in MSE across all 8 dataset–judge pairs, and provide additional analyses on prompt paraphrasing, human annotator disagreement, training size, and cross-generator transfer.
Critical strengths
-
The empirical benchmark of the 13 fixed grading protocols is thorough and genuinely useful for practitioners. Showing that Comparative prompting wins for three judges on Putnam while Debate wins for three on OPC (and backing this up with prompt-paraphrase controls and problem-clustered, Holm-corrected significance tests) convincingly shows that protocol choice matters beyond simple prompt noise.
-
Double-grading 500 solutions with independent human experts is a great sanity check. By testing what happens when you swap the reference human annotator, the authors show that 7 of 8 winning protocols and 93.9% of pairwise rankings remain unchanged, proving their benchmark conclusions aren't just an artifact of one grader's rubric (Appendix O, pp. 35–37; Table 17).
Critical weaknesses
-
There is a test-set leakage issue in how the router checkpoints are selected. Footnote 1 in Appendix R (p. 38) notes that, just like in the main experiments, the router's training epoch is chosen by picking the checkpoint with the lowest MSE on the held-out evaluation fold. Even if no gradients are computed on that fold, using it for early stopping / checkpoint selection means the reported MSE is no longer an unbiased test evaluation. Picking the best single-protocol baseline on the test fold doesn't make this valid, and bootstrapping the chosen predictions afterwards doesn't remove the selection bias. You need a clean three-way split (or inner cross-validation) where the outer test fold is never touched during checkpoint or hyperparameter selection. Also, the text is currently inconsistent between §3.4, Appendix B, and Appendix R on whether routers are trained per candidate generator or pooled together.
-
The baselines don't actually isolate whether problem-dependent routing is doing the heavy lifting. MoG takes a weighted combination of all 13 protocol scores, but it is only compared against individual protocols and an unweighted average. Simply fitting a constant set of simplex weights on the training set (a static learned ensemble) will naturally drop MSE by calibrating score scales and averaging out independent judge errors. Without comparing MoG against learned static weights (and a calibrated version of the single best protocol), we can't tell if the MSE gains come from conditioning the weights on the problem via ModernBERT, or just from doing supervised score ensembling.
Questions for authors
-
Can you confirm whether the held-out fold used to pick the best checkpoint epoch (Appendix R) is the same fold used to report final test MSE? Please provide results where epoch/hyperparameter selection is done strictly on an inner validation split (or via a fixed schedule), clarify whether router training is pooled across candidate families, and ensure bootstrap confidence intervals keep both candidate responses for a given problem in the same cluster.
-
Using the exact same train/test splits and pre-computed score matrices, how does MoG compare against (a) constant simplex weights fitted on the training labels, and (b) linear calibration of the strongest single protocol? How much of the MSE improvement actually comes from making the weights depend on the problem and reference solution?
Thank you. The two baselines you proposed were the most informative experiments in the revision. The main results table is in the general response, where a "setting" means one judge on one benchmark.
Leakage, pooling and clustering (W1, Q1). The stopping epoch is now chosen by 4-fold cross-validation within the training problems; we then retrain and score the untouched test fold. Choosing the epoch on the test fold had lowered MSE by .022 to .046 per setting. Training for a fixed number of epochs with no selection gives results within .03 of the cross-validated ones. With the corrected procedure, routed MoG has lower error than the best fixed protocol in all eight settings, and the difference is significant after Holm correction in four.
The routers in the main experiments and the training-size analysis were each trained on one candidate family. "Pooled" referred to combining the two families' test predictions, and we have clarified §3.4 and Appendices B and R.
All confidence intervals now resample whole problems, so both candidate responses to a problem stay together (10,000 resamples, the same ones for every method). Folds are also assigned by problem for both candidate families.
Static weights and the calibrated best protocol (W2, Q2). Both baselines use the same splits and score matrices as the router. The table counts, out of four judges per benchmark, how each baseline compares with routed MoG after Holm correction.
The third row is the method we now propose as calibrated MoG.
How much of the gain comes from problem-dependent weights? Some on OPC, and little overall. When we replace the router's predicted weights with their average over the training set, the error is unchanged in six settings; the router is better only for the two Qwen judges on OPC. That signal is small next to the effect of calibration. We also trained a router that starts from calibrated MoG and learns per-problem adjustments. Its error is between 5.4% lower and 8.4% higher than calibrated MoG's, and after Holm correction it is not better in any setting and is worse in one.
Calibration versus averaging. You suggested that a static ensemble might win by correcting score scales or by averaging out judge noise, so we tried to separate the two.
Calibration alone does a great deal. The best fixed protocol with a linear correction, at the same grading cost, is significantly better than without it in all eight settings, by 4.6% to 57.1%. The miscalibration can be severe: for DeepSeek-R1 on OPC, the best raw protocol has an MSE of 2.106, which is worse than always predicting the average training label (1.896). Calibrated MoG reaches .817 there.
The learned weights matter too. If we average the 13 protocols uniformly and then calibrate, the result is significantly worse than calibrated MoG in all eight settings.
For averaging, we used the run-to-run variance to estimate the error a protocol would have if it were run infinitely many times. Calibrated MoG beats the best fixed protocol even with infinite runs in seven of eight settings. Against a calibrated single protocol with infinite runs, however, it is significantly better in only one. So beyond calibration, what combining protocols mostly provides is the noise reduction you described, obtained from two or three protocols rather than many repeated runs.
We also tried ridge regression over all 13 protocols, with weights allowed to be negative. It is at least as accurate as calibrated MoG and significantly better in four settings, but it needs all 13 protocols at grading time. We ran this after seeing the calibrated MoG results and say so in the paper.
Finally, calibrated MoG also has the highest Pearson correlation with the expert labels in six of eight settings (.73 to .81, against .67 to .78 for the best fixed protocol). Its gains are larger in MSE than in correlation, and the revision reports both.
To answer your question directly: calibration accounts for most of the gain, the learned weights add to it, and problem-dependent routing adds little. The finding that no protocol is best for every judge and benchmark holds both on the raw scores and after each protocol is calibrated, although the winning protocol changes in five of the eight settings. We report both rankings.