- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Allowing a small LLM to predict when it is incapable before solving a problem before inference helps overcome the trade-off betweencomputational efficiency and output quality. To enhance the prediction accuracy, this paper introduces the Predict-Answer/Act (PA) and Reason-Predict-Reason-Answer/Act (RPRA) paradigms, where an LLM predicts how an external LLM-judge would evaluate its response before generating it. The authors evaluate zero-shot prediction, in-context prediction using model-specific report cards, and prediction after supervised fine-tuning. Their results suggest that large reasoning models can often predict judge-assigned scores directly, while smaller models benefit substantially from report cards or fine-tuning. The study demonstrates the feasibility of using such predictions for model routing.
Critical strengths
- Many LLM baselines are evaluated in this paper, which makes the results and conclusions significantly more convincing.
- The task that this paper aims to solve——deciding when to switch to a larger and more reliable LLM before inference——is indeed meaningful in real-world scenarios, and the PA/RPRA provides an effective solution.
Critical weaknesses
- Outdated empirical coverage and incomplete comparison against recent relevant work.
- As a paper submitted in Sep 2026, there is no any work of this year in its citation. The majority of small LLMs in the experiment are from Llama 3.1/3.2 families. However, many stronger alternatives are already available before submission, such as Qwen3-VL-2B, Qwen3.5-0.8B/2B, Ministral 3-3B/8B. And I believe that it won't take much time to evaluate their performance on zero-shot and contextual settings since it does not need any fine-tuning. The benchmark suite is similarly dated, consisting primarily of datasets released between 2020 and 2024.
- The paper does not adequately discuss closely related pre-inference routing methods. RouteLLM (2024) targets exactly the same problem, and learns query-only routers from human and GPT-4-judge preference labels. Meta-Router (2025) explicitly trains routers using a combination of gold-standard and LLM-as-a-judge supervision. R2-Router, released in February 2026, uses an LLM judge to construct per-model quality-cost curves and predicts these curves before selecting a model and output budget. The paper should therefore clarify its main distinction from these previous approaches in method and performance.
- Limited evidence of cross-distribution generalization and poor scalability to new models.
- Though the idea of introducing report cards sounds plausible, the actual implementation in the paper is benchmark-specific. The authors summarize each model’s modal score on only five predefined domains, and contextual prediction is evaluated on held-out examples drawn from the same datasets used to construct the cards. Consequently, the reported gains may largely reflect in-distribution dataset recognition rather than a general ability to transfer knowledge of model capabilities to new but semantically related tasks. A stronger evaluation would construct a report card using one benchmark and test contextual prediction on a different benchmark from the same domain. (e.g., building the mathematics profile on MATH or AIME 2024 and evaluating it on OlympiadBench).
- The current report-card construction procedure may not scale gracefully when the model pool changes. Currently, the main judge (LLama 3.3 70B) jointly evaluates all candidate responses for each query, so adding a new model may also change previously assigned labels and require recomputing existing report cards.
- Minor writing problems. Though explained in Section 3, the names of the LLMs in the figures are difficult to be understood directly (e.g., L323B is a 3B model while DSQ14B is a 14B model). As a reviewer, I would like to know the model series and scale directly from the figures, as the parameter size of LLMs is crucial in this work. And there are too many similar figures of each dataset in the main paper, which accounts for a really large space with little information density. It would be better using a table to show these experimental results. And the space saved can be used for more detailed explanation of the method, or to showcase some qualitative examples, which are all lacking in the 9 pages. Figure 12 appears ahead of Figure 7-11.
Questions for authors
- In Figures 7-11, zero-shot prediction is evaluated on the training split, whereas contextual and fine-tuned prediction are evaluated on the test split. How can the reported improvements be reliable when different methods are tested on different question set? (The data size of AIME24 and SciCode is only 8 and 26) Could the authors evaluate all methods on the same held-out questions?
- Could the authors show the major difference between RPRA and prior approaches (see Weakness 1)? If prior methods (e.g., RouteLLM) already achieves an excellent performance on the same task, what is the advantage of RPRA?
- Validity of the evaluation target. Now the prediction accuracy measures the agreement with Llama 3.3 70B’s judge-labels. Since the labels are just 'great/ok/bad' categories, could the authors explain whether a higher label-prediction accuracy reflects a better assessment of actual answer quality? I prefer seeing an evaluation on an end-to-end routing system to determine whether the predictions do improve accuracy on these benchmarks.
- If I want the model to infer on a new benchmark or task (e.g.coding, spatial reasoning) not existing on the model card, how can RPRA be transfered to these out-of-domain test sets? (see Weakness 2)
5 comments
Response (2 of 3)
W2a — Report cards are benchmark-specific; gains may reflect in-distribution dataset recognition; suggests building a report card on MATH/AIME and testing on OlympiadBench.
We ran the experiment the reviewer describes, with MATH as the second maths dataset. The maths entry of the report card was built from AIME 2024 only, and contextual prediction was then evaluated on 500 held-out MATH questions that were never used to build any report card.
| Report card built on AIME 2024, tested on MATH (n=500) | Accuracy |
|---|---|
| Llama-3.2-1B | 0.746 |
| Llama-3.1-8B | 0.360 |
| Llama-3.3-70B | 0.702 |
| DeepSeek-R1-Distill-Qwen-14B | 0.582 |
The result matches the in-distribution case. The domain-level prior transfers to a dataset that was never used to build the report card, while instance-level skill stays near zero. So the original numbers are neither a small-sample artefact nor pure dataset recognition. We would note that the transfer is not perfect, and that this is expected. AIME problems are harder than MATH problems, so a report card built on AIME tells a model how it does on hard maths. For the models whose grade is the same on both (bad on both for Llama-3.2-1B, good on both for Llama-3.3-70B and DeepSeek-R1-Distill-Qwen-14B) it carries over exactly. For Llama-3.1-8B, which is bad on AIME but mostly fine on MATH, it carries over the wrong grade. How well the report card transfers tracks how far apart the two datasets are for that model, which is what one would expect of something that carries a domain-level grade. In the updated copy of the paper we will upload before the end of the discussion period, we will add this experiment.
W2b — Joint grading means adding a model may change existing labels and require recomputing report cards.
We concede this for report card construction. The judge grades all candidate responses to a query together, so adding a model to the grading set can shift existing grades. Two mitigations. At inference, a model that has its report card or has been fine-tuned makes its prediction on its own, so changing which other models are deployed does not change its predictions. And the cost of re-grading is small. Adding a model needs, per query, one response from the new model and one new pass of the judge over all the responses, so two extra generations per query however many models are already in the pool, rather than regenerating anything from the existing models. In the updated copy of the paper we will upload before the end of the discussion period, we will state this in the limitations.
W3 — Figure labels (L323B vs DSQ14B), too many similar figures, little information density, Figure 12 ahead of 7-11.
We will be cleaning this up for the updated copy of the paper we will upload before the end of the discussion period. As we have noted to the other reviewers, we will cut Section 2 down and replace the per-dataset figures with one compact presentation of the same results, which also fixes the figure ordering.
Q1 — Zero-shot evaluated on the training split while contextual and fine-tuned are evaluated on the test split; AIME 2024 and SciCode have only 8 and 26 items.
On the split, the zero-shot predictor is untrained, so from its point of view both halves of a uniformly shuffled 75/25 split are unseen data, and its accuracy on the larger half is an unbiased estimate of its accuracy on the smaller one with a tighter interval. We did not re-evaluate on the test half because it cannot change the comparison in expectation, only add noise. In the updated copy of the paper we will upload before the end of the discussion period, we will make this reasoning explicit where the splits are introduced rather than leaving it to a footnote.
On the sizes (the SciCode held-out split is 20 items in our setup), this is the point we think matters most. We therefore reran zero-shot and contextual prediction on two much larger same-domain datasets, MATH and BigCodeBench. Three-way prediction accuracy:
| Dataset | Predictor | n | Zero-shot | Contextual |
|---|---|---|---|---|
| MATH | Llama-3.1-8B | 3,125 | 0.495 | 0.314 |
| MATH | Llama-3.2-1B | 3,122 | 0.106 | 0.718 |
| MATH | Llama-3.2-3B | 3,125 | 0.176 | 0.424 |
| BigCodeBench | Llama-3.1-8B | 285 | 0.235 | 0.333 |
| BigCodeBench | Llama-3.2-1B | 285 | 0.109 | 0.705 |
| BigCodeBench | Llama-3.2-3B | 285 | 0.435 | 0.319 |
On these sample sizes the picture is the same as on the small ones. The report card lifts the two smallest models considerably and does little for the 8B, as in the paper. We will add this to the updated copy of the paper we will upload before the end of the discussion period.
Response (3 of 3)
Q2 — Major difference from RouteLLM and prior approaches; if RouteLLM already performs well, what is the advantage?
Please see W1b. In one sentence, prior routers train a separate model to decide where each query goes, whereas in RPRA each model predicts the grade its own answer would receive and so can make that decision itself, with no separate router model. The two are not directly comparable, since they predict and measure different things, so we do not claim higher routing accuracy than RouteLLM. What we do claim is greater simplicity in an approach that scales more easily to new models. Two things are also specific to RPRA. First, it is the model itself predicting its own ability, not a separate router predicting it from the outside. Second, the target is an LLM judge's own grade. RouteLLM does use GPT-4 as a judge, but only to label which of its two models wins on a query, and that is what its router predicts. RPRA predicts the grade the judge gives the model's response under a rubric that covers accuracy, relevance, completeness, clarity, instruction-following and formatting, so the softer requirements of a response are part of what is predicted rather than a single win or loss.
Q3 — Does higher label-prediction accuracy reflect better assessment of actual answer quality? Prefers an end-to-end routing evaluation.
Two parts. First, whether the labels mean anything. We now have a direct check. For every question in each dataset with a single gold answer, we handed the gold answer itself to the Llama-3.3-70B judge as if it were a model's response, with the reference blanked out of the grading prompt and a correctness-only rubric, and measured how often it is graded correct (great or ok).
| Dataset | n (whole dataset) | Gold answer graded great / ok / bad | Graded correct (great + ok) |
|---|---|---|---|
| MedQA | 12,721 | 10,379 / 1,449 / 893 | 93.0% |
| MMLU-Pro | 11,869 | 8,621 / 1,792 / 1,456 | 87.7% |
| AIME 2024 | 30 | 30 / 0 / 0 | 100.0% |
| SciCode (dev, the only split with reference code) | 15 | 13 / 1 / 1 | 93.3% |
| MATH | 12,500 | 12,342 / 38 / 120 | 99.0% |
| BigCodeBench | 1,139 | 947 / 142 / 50 | 95.6% |
As a control, deliberately misaligned gold answers are graded "bad" 79% of the time on MedQA. The deployed judge also sees the reference answer, so these are a floor. A higher agreement with the judge's label is therefore a higher agreement with correctness, up to the judge's own noise (self-consistency ceiling of about 83% three-way and 94% binary, Section 3.1).
Second, end to end. The paper's Table 2 reports routing outcomes rather than label accuracy, macro-averaged over the five datasets for the fine-tuned predictors. Acc is binary bad-vs-not-bad accuracy, Route% the share sent to a larger model, Bad% the share of all queries kept and then answered badly.
| Fine-tuned predictor | Acc | Route% | Bad% |
|---|---|---|---|
| MobileLLM-0.9B | 0.605 | 98.1 | 0.8 |
| Llama-3.2-1B | 0.772 | 88.7 | 3.2 |
| Llama-3.2-3B | 0.597 | 66.0 | 5.2 |
| Llama-3.1-8B | 0.684 | 39.3 | 8.9 |
The small Llama predictors route 39-89% of queries to a larger model while adding only 3-9% to the bad-answer rate. The per-dataset breakdown for all eleven models, contextual and fine-tuned, is in Appendix M. The report card costs roughly 1,204 extra input tokens per query, the fine-tuned predictor none at inference, which is the compute-versus-quality trade-off the reviewer asks about. In the updated copy of the paper we will upload before the end of the discussion period, we will state these costs alongside Table 2.
Q4 — How does RPRA transfer to a new task not on the report card (e.g., coding, spatial reasoning)?
There are two cases. For a new dataset within a domain that is on the report card, the grounded score transfers within the domain, as shown directly by the AIME-to-MATH test under W2a. For a genuinely new domain with no report card entry, the zero-shot and fine-tuned predictors need no report card at all, and the paper's CommonsenseQA hold-out (Appendix H) reports their behaviour on exactly such a domain, one never used for any report card or any training.
Response (1 of 3)
We thank the reviewer for their time and detailed comments.
W1a — Outdated empirical coverage (no 2026 citations; Llama 3.1/3.2 small models; benchmarks from 2020-2024).
We have run two newer model families through the identical self-prediction pipeline (predicting the Llama-3.3-70B grade, zero-shot and contextual, on the test half of the 75/25 split with up to 200 questions per dataset). These are Apertus-8B (September 2025) and Ministral-3 at 3B, 8B and 14B (December 2025). Three-way prediction accuracy, zero-shot / contextual:
Both families look like the paper's models. Apertus-8B looks like Llama-3.2-1B. Zero-shot it predicts at or below the random-guess level on most datasets, as the paper's smallest models do, and the report card improves it substantially, as it does for the 1B. Ministral-3B and Ministral-8B look like Llama-3.1-8B and the paper's larger models rather than the Llama 3B. They already predict above random zero-shot, and the report card changes their accuracy by a more modest amount, where the Llama 3B still gains substantially from it on several datasets. Ministral-14B is the same, with the report card helping on some datasets and hurting on others. There is no dramatic shift from what we observed on the paper's models, so we do not think the conclusions depend on model release date. In the updated copy of the paper we will upload before the end of the discussion period, we will add these results as an appendix. On the benchmarks, we have added MATH and BigCodeBench (see Q1).
W1b — Missing comparison to RouteLLM, Meta-Router, R2-Router.
This is a genuine omission. The distinction is mechanistic. RouteLLM and Meta-Router each train a separate router that sees only the query and chooses between a stronger and a weaker model, RouteLLM from human pairwise preference data (augmented with gold-labelled or GPT-4-judged data) and Meta-Router from gold-standard and preference-based data combined through a causal-inference framing. R2-Router trains a separate router that models each candidate LLM as a quality-cost curve over output length and chooses both the model and an output budget. RPRA instead has each model predict the grade a judge would give its own answer to the query. The pool of models is still used to build the report cards and the fine-tuning data, but once a model has its report card or has been fine-tuned, it makes its prediction on its own and is functionally independent of the pool. The advantage we claim is only this. No separate router model is needed, since each model can decide for itself whether to answer or hand the query on, which makes the routing itself trivial. The two set-ups are therefore not directly comparable. Those works train and evaluate a router on choosing between models for a query, whereas we measure how well a model predicts the grade a judge would give its own answer and then route on that prediction. The numbers in the two lines of work measure different things, and we have not run them head to head. In the updated copy of the paper we will upload before the end of the discussion period, we will add all three to the related work with this positioning.