acceptodds
Under review as a conference paper at ICLR 2027

Mixture of Graders: Adaptive Strategy Routing for LLM-Based Mathematical Evaluation

Abstract

LLM judges for mathematical reasoning are sensitive to the grading protocol they use. We compare 13 protocols, spanning direct scoring, structured chain-of-thought, and multi-call verification or aggregation, on two new expert-annotated competition-mathematics benchmarks with disjoint problem sources and candidate models: Putnam-AXIOM-Grading (540 problems; 1,080 candidate solutions) and OPC-Grading (500 problems; 1,000 candidate solutions). Across four open-weight judges, no single protocol is best everywhere, and the best choice can flip between benchmarks, a pattern that holds even under controlled prompt paraphrases and Holm-corrected pairwise tests. Building on this, we introduce Mixture of Graders (MoG), a judge-specific router that uses only the problem and reference solution to predict a distribution over grading protocols and combine their scores adaptively for each instance. MoG outperforms the strongest fixed protocol for every open-weight judge on both benchmarks, reducing relative MSE by 6.7 to 27.8% on Putnam-AXIOM-Grading and 3.0 to 13.1% on OPC-Grading. These results show that, for reference-based grading of LLM-generated competition-mathematics solutions, the most useful protocol depends on the instance, and learned routing can exploit this. Code is available at https://anonymous.4open.science/r/mixture-of-graders-01CC/README.md.

Primary area:
datasets and benchmarks
Keywords:
LLM judge, LLM evaluations, chain-of-thought, math reasoning evaluation, prompting, grading, judge reliability, score stability, routing, mixture of models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rQXdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper benchmarks 13 prompting strategies for grading mathematical proofs across four open-weight LLM judges on two new expert-graded datasets (2,080 candidate solutions total, evaluated over 5 runs each). Because the best prompting strategy shifts depending on the judge model and dataset, the authors propose Mixture of Graders (MoG): a ModernBERT router that takes the problem and reference solution and outputs weights to combine all 13 protocol scores. They report that MoG beats the best single protocol in MSE across all 8 dataset–judge pairs, and provide additional analyses on prompt paraphrasing, human annotator disagreement, training size, and cross-generator transfer.

Critical strengths
  • The empirical benchmark of the 13 fixed grading protocols is thorough and genuinely useful for practitioners. Showing that Comparative prompting wins for three judges on Putnam while Debate wins for three on OPC (and backing this up with prompt-paraphrase controls and problem-clustered, Holm-corrected significance tests) convincingly shows that protocol choice matters beyond simple prompt noise.

  • Double-grading 500 solutions with independent human experts is a great sanity check. By testing what happens when you swap the reference human annotator, the authors show that 7 of 8 winning protocols and 93.9% of pairwise rankings remain unchanged, proving their benchmark conclusions aren't just an artifact of one grader's rubric (Appendix O, pp. 35–37; Table 17).

Critical weaknesses
  • There is a test-set leakage issue in how the router checkpoints are selected. Footnote 1 in Appendix R (p. 38) notes that, just like in the main experiments, the router's training epoch is chosen by picking the checkpoint with the lowest MSE on the held-out evaluation fold. Even if no gradients are computed on that fold, using it for early stopping / checkpoint selection means the reported MSE is no longer an unbiased test evaluation. Picking the best single-protocol baseline on the test fold doesn't make this valid, and bootstrapping the chosen predictions afterwards doesn't remove the selection bias. You need a clean three-way split (or inner cross-validation) where the outer test fold is never touched during checkpoint or hyperparameter selection. Also, the text is currently inconsistent between §3.4, Appendix B, and Appendix R on whether routers are trained per candidate generator or pooled together.

  • The baselines don't actually isolate whether problem-dependent routing is doing the heavy lifting. MoG takes a weighted combination of all 13 protocol scores, but it is only compared against individual protocols and an unweighted average. Simply fitting a constant set of simplex weights on the training set (a static learned ensemble) will naturally drop MSE by calibrating score scales and averaging out independent judge errors. Without comparing MoG against learned static weights (and a calibrated version of the single best protocol), we can't tell if the MSE gains come from conditioning the weights on the problem via ModernBERT, or just from doing supervised score ensembling.

Questions for authors
  • Can you confirm whether the held-out fold used to pick the best checkpoint epoch (Appendix R) is the same fold used to report final test MSE? Please provide results where epoch/hyperparameter selection is done strictly on an inner validation split (or via a fixed schedule), clarify whether router training is pooled across candidate families, and ensure bootstrap confidence intervals keep both candidate responses for a given problem in the same cluster.

  • Using the exact same train/test splits and pre-computed score matrices, how does MoG compare against (a) constant simplex weights fitted on the training labels, and (b) linear calibration of the strongest single protocol? How much of the MSE improvement actually comes from making the weights depend on the problem and reference solution?

1 comment

Thank you. The two baselines you proposed were the most informative experiments in the revision. The main results table is in the general response, where a "setting" means one judge on one benchmark.

Leakage, pooling and clustering (W1, Q1). The stopping epoch is now chosen by 4-fold cross-validation within the training problems; we then retrain and score the untouched test fold. Choosing the epoch on the test fold had lowered MSE by .022 to .046 per setting. Training for a fixed number of epochs with no selection gives results within .03 of the cross-validated ones. With the corrected procedure, routed MoG has lower error than the best fixed protocol in all eight settings, and the difference is significant after Holm correction in four.

The routers in the main experiments and the training-size analysis were each trained on one candidate family. "Pooled" referred to combining the two families' test predictions, and we have clarified §3.4 and Appendices B and R.

All confidence intervals now resample whole problems, so both candidate responses to a problem stay together (10,000 resamples, the same ones for every method). Folds are also assigned by problem for both candidate families.

Static weights and the calibrated best protocol (W2, Q2). Both baselines use the same splits and score matrices as the router. The table counts, out of four judges per benchmark, how each baseline compares with routed MoG after Holm correction.

Baseline (fit on training labels only)PutnamOPC
(a) constant simplex weightsno significant difference for any judgeno significant difference for any judge
(b) linear calibration of the best protocolbetter for 2 judges, no difference for 2better for 3 judges, worse for 1
(a) followed by linear calibrationbetter for all 4 judgesbetter for all 4 judges

The third row is the method we now propose as calibrated MoG.

How much of the gain comes from problem-dependent weights? Some on OPC, and little overall. When we replace the router's predicted weights with their average over the training set, the error is unchanged in six settings; the router is better only for the two Qwen judges on OPC. That signal is small next to the effect of calibration. We also trained a router that starts from calibrated MoG and learns per-problem adjustments. Its error is between 5.4% lower and 8.4% higher than calibrated MoG's, and after Holm correction it is not better in any setting and is worse in one.

Calibration versus averaging. You suggested that a static ensemble might win by correcting score scales or by averaging out judge noise, so we tried to separate the two.

Calibration alone does a great deal. The best fixed protocol with a linear correction, at the same grading cost, is significantly better than without it in all eight settings, by 4.6% to 57.1%. The miscalibration can be severe: for DeepSeek-R1 on OPC, the best raw protocol has an MSE of 2.106, which is worse than always predicting the average training label (1.896). Calibrated MoG reaches .817 there.

The learned weights matter too. If we average the 13 protocols uniformly and then calibrate, the result is significantly worse than calibrated MoG in all eight settings.

For averaging, we used the run-to-run variance to estimate the error a protocol would have if it were run infinitely many times. Calibrated MoG beats the best fixed protocol even with infinite runs in seven of eight settings. Against a calibrated single protocol with infinite runs, however, it is significantly better in only one. So beyond calibration, what combining protocols mostly provides is the noise reduction you described, obtained from two or three protocols rather than many repeated runs.

We also tried ridge regression over all 13 protocols, with weights allowed to be negative. It is at least as accurate as calibrated MoG and significantly better in four settings, but it needs all 13 protocols at grading time. We ran this after seeing the calibrated MoG results and say so in the paper.

Finally, calibrated MoG also has the highest Pearson correlation with the expert labels in six of eight settings (.73 to .81, against .67 to .78 for the best fixed protocol). Its gains are larger in MSE than in correlation, and the revision reports both.

To answer your question directly: calibration accounts for most of the gain, the learned weights add to it, and problem-dependent routing adds little. The finding that no protocol is best for every judge and benchmark holds both on the raw scores and after each protocol is calibrated, although the winning protocol changes in five of the eight settings. We report both rankings.

Reviewer_rpDEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates the performance of LLM judges in mathematical reasoning tasks alone, using different commonly used grading protocols. By testing 13 grading protocols (Brief CoT, Self-Critique, and Multi-Judge) on two newly created benchmarks, this work suggests that no single protocol works best. Instead of fixing on a single protocol, this work proposes Mixture of Graders (MoG), which routes to different protocols. Experiments show that MoG can reliably increase grading accuracy.

Critical strengths
  1. Annotations are usually expensive. This work focuses on using an LLM as a judge, which is interesting and timely.
  2. The authors create and share two new benchmarks with the community.
  3. The experiment covers a wide range of grading protocols.
Critical weaknesses
  1. The dataset is meaningful. However, the paper provides little to no discussion of the annotators.
  2. The experiment covers four relatively older LLMs. Adding more modern LLMs could further strengthen the paper's claims.
  3. The router training details could be improved, as there are many BERT variants. It is also unclear whether the BERT router is trained from scratch and how much of the training data is actually truncated.
Questions for authors

See weaknesses

1 comment

Thank you for the review. We respond to each point below and have added the requested details to the paper.

Annotators (W1). Both benchmarks were annotated by two of the authors, who are undergraduate students majoring in mathematics and computer science. They were not compensated beyond authorship. Both used the five-component rubric shown in the Grading Rubric appendix, which is the same rubric given to the LLM judges, and they graded without seeing any judge outputs.

To measure agreement, both annotators independently graded 500 solutions, 250 from each benchmark. The quadratic-weighted κ is 0.857 on Putnam and 0.815 on OPC, and the mean absolute difference between the two annotators is 0.540 points on the 0–5 scale. Appendix O examines the disagreements. It also shows that our conclusions do not depend on whose grades are used: when the reference annotator is swapped, the winning protocol stays the same in seven of the eight judge–benchmark settings, and 93.9% of pairwise protocol rankings are unchanged. These details are now in §3.1.

Newer judges (W2). Our four judges cover two model families (Qwen and Llama), two sizes, and one reasoning-distilled model. The finding that the best protocol depends on the judge and the benchmark holds for each of them, including under the prompt-paraphrase and annotator-swap checks. Adding one more judge to the full study means 13 protocols × 5 runs × 2,080 solutions, about 135,000 gradings, which we could not complete during the discussion period. We state this scope in the Limitations.

The revised method does make it much cheaper to move to a new judge. Calibrated MoG combines two or three protocols and is fit by least squares on roughly 50 to 100 labeled problems, so a new judge needs neither a router nor the full 13-protocol sweep.

Router details (W3). The router is answerdotai/ModernBERT-large (395M parameters), fine-tuned from the pretrained checkpoint. Only the final 13-way linear layer is newly initialized. The phrase "from scratch" in Appendix R meant that each configuration gets its own fine-tuning run, and we have reworded it. The router reads the problem statement and the reference solution, never the candidate solution.

Training uses mean pooling, dropout 0.1, and a softmax over the 13 protocols, with the mixture-MSE loss from §3.4. The optimizer is AdamW with a learning rate of 2e-5 for the encoder and 1e-3 for the linear layer, weight decay 0.01, a linear schedule with 10% warmup, and gradient clipping at 1.0. We use batch size 16 and train for 8 epochs, the first 2 with the encoder frozen, with seed 0.

On truncation, these are the input lengths under the ModernBERT tokenizer:

BenchmarkMedian length (tokens)90th percentileLonger than 1,024Longer than 2,048
Putnam4878996.7%none
OPC7231,78028.6%7.8%

The routers use a 2,048-token limit on Putnam, where nothing is truncated, and 1,024 on OPC. The revised main method, calibrated MoG, has no text encoder, so truncation does not affect it. The appendix now includes a router-configuration table and these statistics.

Reviewer_1JurOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how different grading protocols affect the accuracy and stability of LLM-based grading of mathematical solutions. The authors compare 13 grading protocols across four open-weight models on two expert-annotated datasets and propose Mixture of Graders (MoG), which assigns weights to the protocols based on the problem and reference solution. The reported results show that MoG outperforms the best fixed protocol on both datasets.

Critical strengths
  1. The paper provides a useful, controlled comparison of mathematical grading protocols. Partial-credit grading requires assessing the reasoning beyond final-answer correctness. Comparing 13 protocols across four judges and two datasets, with problem-clustered statistical analysis, Holm correction, and controlled prompt paraphrasing, supports the conclusion that no fixed protocol is universally optimal within the tested settings. This empirical finding is a substantive contribution because it shows how protocol choice can affect evaluations of mathematical reasoning.

  2. The annotation-reliability analysis strengthens the fixed-protocol findings. Independent double-grading of 500 solutions and the analysis of ranking sensitivity to annotator choice provide evidence that the broad comparisons are not artifacts of one annotator's scoring preferences. The disagreement audit also identifies errors by both humans and models. These checks increase confidence in the paper's empirical comparison despite uncertainty in individual partial-credit labels.

Critical weaknesses
  1. The MoG evaluation is not independent of checkpoint selection, and key experimental descriptions are inconsistent. Appendix R explicitly states that, as in the main runs, checkpoint epochs are selected by MSE on the held-out fold used for evaluation. This can make the reported gains optimistic and also affects the training-size conclusions. In addition, the descriptions of fixed-baseline selection and separate versus pooled Putnam router training need reconciliation (see Question 3). These issues limit confidence in the reported routing gains; checkpoint selection should use an internal validation split, followed by evaluation on independent test folds under a clearly specified protocol.

  2. A learned global fixed-weight ensemble baseline is missing. Beating the best single protocol and uniform averaging does not establish that MoG benefits from adapting weights to each problem; the improvement could instead come from learning better global weights. A fixed-weight convex combination independent of the input should be trained on the same data and evaluated on the same independent test folds. Without this comparison, the evidence does not isolate the benefit of problem-dependent routing, which is the central methodological claim.

Questions for authors
  1. I would like to see MoG's improvement over the best fixed grading protocol on independent test folds that have not been used for checkpoint selection, including a repeat of the training-size analysis under this protocol.

  2. I would like to see a direct comparison between MoG and a learned global fixed-weight convex combination, using identical training data and independent test sets. It would also be useful to report how performance changes when routing weights are randomly shuffled across test problems.

  3. Table 9 attributes the MSE difference between 0.829 and 0.812 to fold-wise validation selection, whereas Section 3.4 states that the fixed protocol is selected on the evaluated held-out fold itself. Under the latter procedure, with identical instances, scores, and aggregation weights, fold-wise selection cannot perform worse than always using Comparative. Does Table 9 use a separate validation set, and which selection rule applies to each reported comparison? Please also clarify why Section 3.4 describes separate Putnam routers for each candidate response set while Appendix R pools both response sets for training and describes its recipe as identical to the main runs. Specify the configurations for the main experiments and the training-size ablation.

1 comment

Thank you for the careful reading. We ran each experiment you asked for. The main results table is in the general response, where a "setting" means one judge on one benchmark.

Independent test folds (W1, Q1). The router's stopping epoch is now chosen by 4-fold cross-validation within the training problems. We then retrain on all training problems and score the test fold once. To measure how much the old procedure mattered, we ran both on the same folds: choosing the epoch on the test fold had lowered MSE by .022 to .046, depending on the setting. We also trained for a fixed 8 epochs with no selection at all, which lands within .03 of the cross-validated result everywhere.

Under the corrected procedure, routed MoG has lower error than the best fixed protocol (itself chosen on training data) in all eight settings, by 1.5% to 24.6%. The difference is significant after Holm correction in four: both Qwen judges on Putnam, and Qwen2.5-72B and DeepSeek-R1 on OPC.

Training-size analysis. We repeated it for Qwen2.5-72B with the fixed 8-epoch schedule, so no checkpoint is selected at any point. N is the number of training problems per fold, and each entry is MSE averaged over random subsets of that size.

NPutnam: best fixed / routed MoG / calibrated MoGOPC: best fixed / routed MoG / calibrated MoG
10.954 / .745 / .8671.345 / 1.083 / 1.116
25.912 / .695 / .7351.173 / 1.070 / .938
50.903 / .665 / .6661.105 / 1.050 / .896
100.893 / .649 / .6321.105 / .992 / .871
200.862 / .626 / .6111.105 / .970 / .854
all.845 / .633 / .6011.105 / .958 / .845

Routed MoG has lower error than the best fixed protocol at every training size, and its confidence interval excludes zero in 28 of the 30 subset runs. The conclusion of the original analysis therefore does not depend on checkpoint selection.

The table also shows when each version of the method is preferable. With very few labels (10 to 25 problems on Putnam, 10 on OPC), routed MoG is the better choice, because the linear correction in calibrated MoG overfits. With more labels, calibrated MoG is as good or better: it is significantly better on OPC from 25 problems, and better on average on Putnam from 100.

Fixed-weight ensemble and shuffled weights (W2, Q2). We added both, using the same training data and folds as the router.

The fixed-weight ensemble, with one learned weight per protocol, matches routed MoG in every setting. The two never differ by more than .021 MSE, and no difference is significant.

For the shuffling test, we permuted the router's predicted weights across test problems 100 times. This raises MSE by less than .035 in six settings and by .11 to .12 in the other two, which are the two Qwen judges on OPC. Replacing the predicted weights with their average gives the same picture: no change in six settings, and a loss of .04 to .05 for the Qwen judges on OPC. So the router does pick up problem-specific signal on OPC, which draws on 13 different competitions, but little on Putnam.

Your fixed-weight baseline is also what led us to the revised method. Once the weights are fixed, it is natural to add a linear correction to the output, and that correction matters more than routing: the resulting calibrated MoG is significantly better than routed MoG in all eight settings. A router trained on top of calibrated MoG is not significantly better than it in any setting.

Table 9 and router configuration (Q3). Thank you for catching the mismatch. The two numbers come from two ways of choosing the fixed baseline. Using Comparative on every fold gives .812. Table 9 instead chose the best protocol separately for each fold, using that fold's training data; No-CoT was chosen on one fold, and the result is .829. A choice made on training data can do worse on the test folds than always using Comparative, which is what happened. Your inequality applies when the choice is made on the test fold itself. That rule gives .793, which is below .812 as you argued. In the revision every comparison chooses the fixed baseline on training data, and each caption says so.

On the routers: each one, in both the main experiments and the training-size analysis, was trained on a single candidate set. "Pooled" referred to combining the test predictions from the two sets. The training-size analysis therefore trained 250 routers per benchmark rather than 125, and we have corrected §3.4 and Appendix R. In the new runs the two candidate sets share one problem-level split.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.