acceptodds
Under review as a conference paper at ICLR 2027

RPRA: Predicting an LLM-Judge for Efficient but Performant Inference

Abstract

Large language models (LLMs) face a fundamental trade-off between computational efficiency (e.g., number of parameters) and output quality, especially when deployed on computationally limited devices such as phones or laptops. One way to address this challenge is by following the example of humans and have models ask for help when they believe they are incapable of solving a problem on their own; we can overcome this trade-off by allowing smaller models to respond to queries when they believe they can provide good responses, and deferring to larger models when they do not believe they can. To this end, in this paper, we investigate the viability of Predict-Answer/Act (PA) and Reason-Predict-Reason-Answer/Act (RPRA) paradigms where models predict—prior to responding—how an LLM judge would score their output. We evaluate three approaches: zero-shot prediction, prediction using an in-context report card, and supervised fine-tuning. Our results show that larger models (particularly reasoning models) perform well when predicting generic LLM judges zero-shot, while smaller models can reliably predict such judges well after being fine-tuned or provided with an in-context report card. Altogether, both approaches can substantially improve the prediction accuracy of smaller models, with report cards and fine-tuning achieving mean improvements of up to 55% and 52% across datasets, respectively. These findings suggest that models can learn to predict their own performance limitations, paving the way for more efficient and self-aware AI systems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic AI, Large Language Models, LLM-as-a-Judge, Self-Reflective Models, Supervised Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zViVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Allowing a small LLM to predict when it is incapable before solving a problem before inference helps overcome the trade-off betweencomputational efficiency and output quality. To enhance the prediction accuracy, this paper introduces the Predict-Answer/Act (PA) and Reason-Predict-Reason-Answer/Act (RPRA) paradigms, where an LLM predicts how an external LLM-judge would evaluate its response before generating it. The authors evaluate zero-shot prediction, in-context prediction using model-specific report cards, and prediction after supervised fine-tuning. Their results suggest that large reasoning models can often predict judge-assigned scores directly, while smaller models benefit substantially from report cards or fine-tuning. The study demonstrates the feasibility of using such predictions for model routing.

Critical strengths
  1. Many LLM baselines are evaluated in this paper, which makes the results and conclusions significantly more convincing.
  2. The task that this paper aims to solve——deciding when to switch to a larger and more reliable LLM before inference——is indeed meaningful in real-world scenarios, and the PA/RPRA provides an effective solution.
Critical weaknesses
  1. Outdated empirical coverage and incomplete comparison against recent relevant work.
  • As a paper submitted in Sep 2026, there is no any work of this year in its citation. The majority of small LLMs in the experiment are from Llama 3.1/3.2 families. However, many stronger alternatives are already available before submission, such as Qwen3-VL-2B, Qwen3.5-0.8B/2B, Ministral 3-3B/8B. And I believe that it won't take much time to evaluate their performance on zero-shot and contextual settings since it does not need any fine-tuning. The benchmark suite is similarly dated, consisting primarily of datasets released between 2020 and 2024.
  • The paper does not adequately discuss closely related pre-inference routing methods. RouteLLM (2024) targets exactly the same problem, and learns query-only routers from human and GPT-4-judge preference labels. Meta-Router (2025) explicitly trains routers using a combination of gold-standard and LLM-as-a-judge supervision. R2-Router, released in February 2026, uses an LLM judge to construct per-model quality-cost curves and predicts these curves before selecting a model and output budget. The paper should therefore clarify its main distinction from these previous approaches in method and performance.
  1. Limited evidence of cross-distribution generalization and poor scalability to new models.
  • Though the idea of introducing report cards sounds plausible, the actual implementation in the paper is benchmark-specific. The authors summarize each model’s modal score on only five predefined domains, and contextual prediction is evaluated on held-out examples drawn from the same datasets used to construct the cards. Consequently, the reported gains may largely reflect in-distribution dataset recognition rather than a general ability to transfer knowledge of model capabilities to new but semantically related tasks. A stronger evaluation would construct a report card using one benchmark and test contextual prediction on a different benchmark from the same domain. (e.g., building the mathematics profile on MATH or AIME 2024 and evaluating it on OlympiadBench).
  • The current report-card construction procedure may not scale gracefully when the model pool changes. Currently, the main judge (LLama 3.3 70B) jointly evaluates all candidate responses for each query, so adding a new model may also change previously assigned labels and require recomputing existing report cards.
  1. Minor writing problems. Though explained in Section 3, the names of the LLMs in the figures are difficult to be understood directly (e.g., L323B is a 3B model while DSQ14B is a 14B model). As a reviewer, I would like to know the model series and scale directly from the figures, as the parameter size of LLMs is crucial in this work. And there are too many similar figures of each dataset in the main paper, which accounts for a really large space with little information density. It would be better using a table to show these experimental results. And the space saved can be used for more detailed explanation of the method, or to showcase some qualitative examples, which are all lacking in the 9 pages. Figure 12 appears ahead of Figure 7-11.
Questions for authors
  1. In Figures 7-11, zero-shot prediction is evaluated on the training split, whereas contextual and fine-tuned prediction are evaluated on the test split. How can the reported improvements be reliable when different methods are tested on different question set? (The data size of AIME24 and SciCode is only 8 and 26) Could the authors evaluate all methods on the same held-out questions?
  2. Could the authors show the major difference between RPRA and prior approaches (see Weakness 1)? If prior methods (e.g., RouteLLM) already achieves an excellent performance on the same task, what is the advantage of RPRA?
  3. Validity of the evaluation target. Now the prediction accuracy measures the agreement with Llama 3.3 70B’s judge-labels. Since the labels are just 'great/ok/bad' categories, could the authors explain whether a higher label-prediction accuracy reflects a better assessment of actual answer quality? I prefer seeing an evaluation on an end-to-end routing system to determine whether the predictions do improve accuracy on these benchmarks.
  4. If I want the model to infer on a new benchmark or task (e.g.coding, spatial reasoning) not existing on the model card, how can RPRA be transfered to these out-of-domain test sets? (see Weakness 2)
5 comments
AuthorsOpenReview ↗

Response (1 of 3)

We thank the reviewer for their time and detailed comments.

W1a — Outdated empirical coverage (no 2026 citations; Llama 3.1/3.2 small models; benchmarks from 2020-2024).

We have run two newer model families through the identical self-prediction pipeline (predicting the Llama-3.3-70B grade, zero-shot and contextual, on the test half of the 75/25 split with up to 200 questions per dataset). These are Apertus-8B (September 2025) and Ministral-3 at 3B, 8B and 14B (December 2025). Three-way prediction accuracy, zero-shot / contextual:

ModelMATHGSM8KMMLU-ProMedQABigCodeBenchCommonsenseQA (hold-out)
Apertus-8B.161 / .244.205 / .362.165 / .364.141 / .641.495 / .293.375 / .321
Ministral-3B.665 / .670.590 / .550.380 / .360.360 / .410.410 / .430.395 / .410
Ministral-8B.545 / .595.385 / .395.415 / .480.425 / .440.475 / .290.565 / .620
Ministral-14B.400 / .490.465 / .140.520 / .340.400 / .435.335 / .165.615 / .105

Both families look like the paper's models. Apertus-8B looks like Llama-3.2-1B. Zero-shot it predicts at or below the random-guess level on most datasets, as the paper's smallest models do, and the report card improves it substantially, as it does for the 1B. Ministral-3B and Ministral-8B look like Llama-3.1-8B and the paper's larger models rather than the Llama 3B. They already predict above random zero-shot, and the report card changes their accuracy by a more modest amount, where the Llama 3B still gains substantially from it on several datasets. Ministral-14B is the same, with the report card helping on some datasets and hurting on others. There is no dramatic shift from what we observed on the paper's models, so we do not think the conclusions depend on model release date. In the updated copy of the paper we will upload before the end of the discussion period, we will add these results as an appendix. On the benchmarks, we have added MATH and BigCodeBench (see Q1).

W1b — Missing comparison to RouteLLM, Meta-Router, R2-Router.

This is a genuine omission. The distinction is mechanistic. RouteLLM and Meta-Router each train a separate router that sees only the query and chooses between a stronger and a weaker model, RouteLLM from human pairwise preference data (augmented with gold-labelled or GPT-4-judged data) and Meta-Router from gold-standard and preference-based data combined through a causal-inference framing. R2-Router trains a separate router that models each candidate LLM as a quality-cost curve over output length and chooses both the model and an output budget. RPRA instead has each model predict the grade a judge would give its own answer to the query. The pool of models is still used to build the report cards and the fine-tuning data, but once a model has its report card or has been fine-tuned, it makes its prediction on its own and is functionally independent of the pool. The advantage we claim is only this. No separate router model is needed, since each model can decide for itself whether to answer or hand the query on, which makes the routing itself trivial. The two set-ups are therefore not directly comparable. Those works train and evaluate a router on choosing between models for a query, whereas we measure how well a model predicts the grade a judge would give its own answer and then route on that prediction. The numbers in the two lines of work measure different things, and we have not run them head to head. In the updated copy of the paper we will upload before the end of the discussion period, we will add all three to the related work with this positioning.

AuthorsOpenReview ↗

Response (2 of 3)

W2a — Report cards are benchmark-specific; gains may reflect in-distribution dataset recognition; suggests building a report card on MATH/AIME and testing on OlympiadBench.

We ran the experiment the reviewer describes, with MATH as the second maths dataset. The maths entry of the report card was built from AIME 2024 only, and contextual prediction was then evaluated on 500 held-out MATH questions that were never used to build any report card.

Report card built on AIME 2024, tested on MATH (n=500)Accuracy
Llama-3.2-1B0.746
Llama-3.1-8B0.360
Llama-3.3-70B0.702
DeepSeek-R1-Distill-Qwen-14B0.582

The result matches the in-distribution case. The domain-level prior transfers to a dataset that was never used to build the report card, while instance-level skill stays near zero. So the original numbers are neither a small-sample artefact nor pure dataset recognition. We would note that the transfer is not perfect, and that this is expected. AIME problems are harder than MATH problems, so a report card built on AIME tells a model how it does on hard maths. For the models whose grade is the same on both (bad on both for Llama-3.2-1B, good on both for Llama-3.3-70B and DeepSeek-R1-Distill-Qwen-14B) it carries over exactly. For Llama-3.1-8B, which is bad on AIME but mostly fine on MATH, it carries over the wrong grade. How well the report card transfers tracks how far apart the two datasets are for that model, which is what one would expect of something that carries a domain-level grade. In the updated copy of the paper we will upload before the end of the discussion period, we will add this experiment.

W2b — Joint grading means adding a model may change existing labels and require recomputing report cards.

We concede this for report card construction. The judge grades all candidate responses to a query together, so adding a model to the grading set can shift existing grades. Two mitigations. At inference, a model that has its report card or has been fine-tuned makes its prediction on its own, so changing which other models are deployed does not change its predictions. And the cost of re-grading is small. Adding a model needs, per query, one response from the new model and one new pass of the judge over all the responses, so two extra generations per query however many models are already in the pool, rather than regenerating anything from the existing models. In the updated copy of the paper we will upload before the end of the discussion period, we will state this in the limitations.

W3 — Figure labels (L323B vs DSQ14B), too many similar figures, little information density, Figure 12 ahead of 7-11.

We will be cleaning this up for the updated copy of the paper we will upload before the end of the discussion period. As we have noted to the other reviewers, we will cut Section 2 down and replace the per-dataset figures with one compact presentation of the same results, which also fixes the figure ordering.

Q1 — Zero-shot evaluated on the training split while contextual and fine-tuned are evaluated on the test split; AIME 2024 and SciCode have only 8 and 26 items.

On the split, the zero-shot predictor is untrained, so from its point of view both halves of a uniformly shuffled 75/25 split are unseen data, and its accuracy on the larger half is an unbiased estimate of its accuracy on the smaller one with a tighter interval. We did not re-evaluate on the test half because it cannot change the comparison in expectation, only add noise. In the updated copy of the paper we will upload before the end of the discussion period, we will make this reasoning explicit where the splits are introduced rather than leaving it to a footnote.

On the sizes (the SciCode held-out split is 20 items in our setup), this is the point we think matters most. We therefore reran zero-shot and contextual prediction on two much larger same-domain datasets, MATH and BigCodeBench. Three-way prediction accuracy:

DatasetPredictornZero-shotContextual
MATHLlama-3.1-8B3,1250.4950.314
MATHLlama-3.2-1B3,1220.1060.718
MATHLlama-3.2-3B3,1250.1760.424
BigCodeBenchLlama-3.1-8B2850.2350.333
BigCodeBenchLlama-3.2-1B2850.1090.705
BigCodeBenchLlama-3.2-3B2850.4350.319

On these sample sizes the picture is the same as on the small ones. The report card lifts the two smallest models considerably and does little for the 8B, as in the paper. We will add this to the updated copy of the paper we will upload before the end of the discussion period.

AuthorsOpenReview ↗

Response (3 of 3)

Q2 — Major difference from RouteLLM and prior approaches; if RouteLLM already performs well, what is the advantage?

Please see W1b. In one sentence, prior routers train a separate model to decide where each query goes, whereas in RPRA each model predicts the grade its own answer would receive and so can make that decision itself, with no separate router model. The two are not directly comparable, since they predict and measure different things, so we do not claim higher routing accuracy than RouteLLM. What we do claim is greater simplicity in an approach that scales more easily to new models. Two things are also specific to RPRA. First, it is the model itself predicting its own ability, not a separate router predicting it from the outside. Second, the target is an LLM judge's own grade. RouteLLM does use GPT-4 as a judge, but only to label which of its two models wins on a query, and that is what its router predicts. RPRA predicts the grade the judge gives the model's response under a rubric that covers accuracy, relevance, completeness, clarity, instruction-following and formatting, so the softer requirements of a response are part of what is predicted rather than a single win or loss.

Q3 — Does higher label-prediction accuracy reflect better assessment of actual answer quality? Prefers an end-to-end routing evaluation.

Two parts. First, whether the labels mean anything. We now have a direct check. For every question in each dataset with a single gold answer, we handed the gold answer itself to the Llama-3.3-70B judge as if it were a model's response, with the reference blanked out of the grading prompt and a correctness-only rubric, and measured how often it is graded correct (great or ok).

Datasetn (whole dataset)Gold answer graded great / ok / badGraded correct (great + ok)
MedQA12,72110,379 / 1,449 / 89393.0%
MMLU-Pro11,8698,621 / 1,792 / 1,45687.7%
AIME 20243030 / 0 / 0100.0%
SciCode (dev, the only split with reference code)1513 / 1 / 193.3%
MATH12,50012,342 / 38 / 12099.0%
BigCodeBench1,139947 / 142 / 5095.6%

As a control, deliberately misaligned gold answers are graded "bad" 79% of the time on MedQA. The deployed judge also sees the reference answer, so these are a floor. A higher agreement with the judge's label is therefore a higher agreement with correctness, up to the judge's own noise (self-consistency ceiling of about 83% three-way and 94% binary, Section 3.1).

Second, end to end. The paper's Table 2 reports routing outcomes rather than label accuracy, macro-averaged over the five datasets for the fine-tuned predictors. Acc is binary bad-vs-not-bad accuracy, Route% the share sent to a larger model, Bad% the share of all queries kept and then answered badly.

Fine-tuned predictorAccRoute%Bad%
MobileLLM-0.9B0.60598.10.8
Llama-3.2-1B0.77288.73.2
Llama-3.2-3B0.59766.05.2
Llama-3.1-8B0.68439.38.9

The small Llama predictors route 39-89% of queries to a larger model while adding only 3-9% to the bad-answer rate. The per-dataset breakdown for all eleven models, contextual and fine-tuned, is in Appendix M. The report card costs roughly 1,204 extra input tokens per query, the fine-tuned predictor none at inference, which is the compute-versus-quality trade-off the reviewer asks about. In the updated copy of the paper we will upload before the end of the discussion period, we will state these costs alongside Table 2.

Q4 — How does RPRA transfer to a new task not on the report card (e.g., coding, spatial reasoning)?

There are two cases. For a new dataset within a domain that is on the report card, the grounded score transfers within the domain, as shown directly by the AIME-to-MATH test under W2a. For a genuinely new domain with no report card entry, the zero-shot and fine-tuned predictors need no report card at all, and the paper's CommonsenseQA hold-out (Appendix H) reports their behaviour on exactly such a domain, one never used for any report card or any training.

Reviewer_nY5qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed a RPRA framework to improve the efficiency. It contains a small model and a large LLM, while the small model would be used to justify whether the answer could be bad or good, then route the bad ones to large LLMs.

The motivation here is it would be more accurate to predict the quality of a response rather than producing the answer.

Critical strengths
  1. The motivation is practical and easy to understand. Asking a small model to predict before generation whether it can answer a query is a sensible routing idea.
  2. The paper tests 11 models across five main benchmarks and includes helpful appendix ablations on alternative judges, prompt templates, and rubric variations.
Critical weaknesses
  1. The paper layout is still hard to follow. Figure 6 appears on page 5, Figure 12 jumps to page 6, and Figures 7-11 only appear on page 7. Section 2 also spends a full page repeating basic transformer definitions that are not used later.
  2. It is still unclear whether the Llama 3.3 70B grades reflect true task accuracy. Section 3.1 only checks whether the judge agrees with itself across five runs, which measures consistency rather than correctness.
  3. On small models, the reported gains look driven by constant "bad" predictions rather than instance-level self-assessment.
  4. End-to-end routing and reproducibility is incomplete.
Questions for authors

In weakness.

4 comments
AuthorsOpenReview ↗

Response (1 of 2)

We thank the reviewer for their time and thoughtful comments.

W1 — "The paper layout is still hard to follow. Figure 6 appears on page 5, Figure 12 jumps to page 6, and Figures 7-11 only appear on page 7. Section 2 also spends a full page repeating basic transformer definitions that are not used later."

Agreed on both counts. In the updated copy of the paper we will upload before the end of the discussion period, we will cut Section 2 to a compact preliminaries block without the transformer notation, and replace Figures 7-11 with one compact presentation of the same results, which removes the ordering problem at its source.

W2 — "It is still unclear whether the Llama 3.3 70B grades reflect true task accuracy. Section 3.1 only checks whether the judge agrees with itself across five runs, which measures consistency rather than correctness."

The reviewer is right that the five-run check measures consistency and not correctness. Note that the correctness of LLM judges for this purpose is well established in the work we build on. Zheng et al. (2023) show single-LLM judges reaching Spearman correlations of roughly 0.8 with aggregated human preferences across diverse tasks, with G-Eval (Liu et al., 2023), GPTScore (Fu et al., 2024) and Agent-as-a-Judge (Zhuge et al., 2025) reporting similar agreement with humans, and Self-Taught Evaluators (Wang et al., 2024) building a strong judge on exactly this Llama 3 70B family. Llama 3.3 70B is a well-studied model and we chose it for that reason. That said, we have now also run a direct correctness check on the judge.

For every question in each dataset that has a single gold answer, we handed the gold answer itself to the Llama-3.3-70B judge as if it were a model's response, with the reference answer blanked out of the grading prompt and a correctness-only rubric. A valid grader should call a known-correct answer correct almost every time. This is a deliberately stringent version of the test, since the judge as deployed in the paper also sees the reference answer. The numbers below are therefore a floor on the deployed judge's agreement with ground truth (note also that the judge grades more amorphous properties of an answer, such as clarity, which cannot be checked in this way).

Datasetn (whole dataset)Gold answer graded great / ok / badGraded correct (great + ok)
MedQA12,72110,379 / 1,449 / 89393.0%
MMLU-Pro11,8698,621 / 1,792 / 1,45687.7%
AIME 20243030 / 0 / 0100.0%
SciCode (dev, the only split with reference code)1513 / 1 / 193.3%
MATH12,50012,342 / 38 / 12099.0%
BigCodeBench1,139947 / 142 / 5095.6%

As a control, the same pipeline with the gold answers deliberately misaligned to the wrong questions (so query 1 is answered by feeding the gold answer from query 2) is graded "bad" 79% of the time on MedQA (versus 7% when aligned), so the judge is discriminating correctness and not just rewarding the form of an answer. We also audited the 893 MedQA "bad" cases. Every one has a non-empty, correctly paired gold answer injected, and in the rationales we read the judge disputes the answer on medical grounds rather than failing to parse it. So the 7% is the judge being wrong, not the pipeline. LongFact has no single reference answer, so this kind of check cannot be carried out there.

We read this as the great/ok/bad labels tracking real correctness on the datasets where that can be checked. Where the judge disagrees with the gold answer it does so by under-crediting a correct answer. In the updated copy of the paper we will upload before the end of the discussion period, we will add this as a subsection alongside the self-consistency check in Section 3.1.

AuthorsOpenReview ↗

Response (2 of 2)

W3 — "On small models, the reported gains look driven by constant 'bad' predictions rather than instance-level self-assessment."

For the two smallest models this is correct, and we acquiesce to the point. MobileLLM-0.9B and Llama-3.2-1B are graded "bad" on most of what they answer, and their predictors learn to say so. We would add, though, that this is not a failure of the prediction. A model that is nearly always bad and predicts "bad" nearly always has understood its own ability correctly, and that is what the gain in Table 1 for these two models reflects. The reviewer is likely identifying the downstream implications. A predictor that says "bad" to everything is useless as a router, because it routes everything away.

The paper's routing table (Table 2) is the direct measure of this, since Route% is the fraction of queries a predictor called "bad":

Fine-tuned predictorAccRoute%Bad%
MobileLLM-0.9B0.60598.10.8
Llama-3.2-1B0.77288.73.2
Llama-3.2-3B0.59766.05.2
Llama-3.1-8B0.68439.38.9

MobileLLM-0.9B and Llama-3.2-1B route 98% and 89% of queries away, which is the constant-"bad" behaviour the reviewer describes. However, this stops quite quickly with Llama-3.2-3B routing 66% and Llama-3.1-8B 39%, so these predictors are saying "not bad" to a third and to three fifths of queries respectively, which a constant prediction cannot do. In the updated copy of the paper we will upload before the end of the discussion period, we will make this distinction more explicit. For the two smallest models the gain in prediction accuracy is real but comes from a correct constant prediction, and it is the 3B and 8B models that are useful for routing, as we note.

W4 — "End-to-end routing and reproducibility is incomplete."

On routing, the paper's Table 2 reports the routing metrics for the four fine-tuned predictors, macro-averaged over the five datasets. Acc is the binary bad-vs-not-bad prediction accuracy, Route% the share of queries the predictor sends to a larger model, and Bad% the share of all queries the model keeps and then answers badly.

Fine-tuned predictorAccRoute%Bad%
MobileLLM-0.9B0.60598.10.8
Llama-3.2-1B0.77288.73.2
Llama-3.2-3B0.59766.05.2
Llama-3.1-8B0.68439.38.9

The small Llama predictors route 39-89% of queries to a larger model while adding only 3-9% to the bad-answer rate. The full per-dataset routing for all eleven models, in both the contextual and fine-tuned settings, is in Appendix M (Tables 6-10). On cost, the report card adds roughly 1,204 input tokens per query in our measurement. The fine-tuned predictor bakes this into the weights and adds no extra tokens at inference. In the updated copy of the paper we will upload before the end of the discussion period, we will state these costs alongside Table 2.

The paper's Table 1 reports, per model and dataset, how much fine-tuning improves prediction accuracy over zero-shot, but only for the four small models. The contextual counterpart for all eleven models is Table 3 in Appendix B. We reproduce it here since it is the natural companion, and in the updated copy of the paper we will upload before the end of the discussion period, we will move it to the main text.

ModelMedQALongFactAIME 2024MMLU-ProSciCodeMean
MobileLLM-0.9B−0.10+0.27−0.15−0.11−0.43−0.10
Llama-3.2-1B+0.28+0.12+0.91+0.55+0.90+0.55
Llama-3.2-3B−0.01−0.12+0.35+0.24−0.12+0.07
Llama-3.1-8B+0.22−0.05−0.220.00+0.030.00
DeepSeek-R1-Distill-Qwen-14B+0.14+0.07−0.17+0.07−0.02+0.02
GPT-OSS-20B0.00−0.13+0.16−0.01−0.08−0.01
DeepSeek-R1-Distill-Qwen-32B+0.03+0.09+0.75+0.02−0.13+0.15
Llama-3.3-70B+0.59+0.67−0.03+0.24+0.38+0.37
DeepSeek-R1-Distill-Llama-70B+0.14+0.21+0.53+0.10−0.22+0.15
Llama-4-Scout+0.06+0.08+0.01+0.13+0.28+0.11
GPT-OSS-120B+0.04+0.18+0.05−0.05+0.18+0.08

On reproducibility, the redacted block in the submission is the release link, blacked out for double-blind review. We will be releasing the full source code, and in the updated copy of the paper we will upload before the end of the discussion period, that block will link to an anonymised repository containing it.

Thanks for your response.

My concern on W1 has been resolved. But the concern on W2 and W3 are not solved. Besides that, the anonymised code has not been uploaded.

I will maintain my score.

AuthorsOpenReview ↗

Reply to Official Comment by Reviewer nY5q

Can the reviewer please clarify in what way their concerns regarding W2 and W3 are not resolved?

Regarding W2: we have shown quite clearly in Section 3.2 that the judge's grades track real correctness, with gold answers graded correct on 87.7% to 100% of each dataset tested. This was already strongly supported in the initial submission by the extensive prior work on LLM judges and by Llama 3.3 70B being a well-studied model.

Regarding W3: the paper is transparent in Section 6 that the smallest models are indeed driven by this. For very small models, this is the desired behaviour, and their gain in prediction accuracy is real. For the 3B and 8B models, this fades and the routing benefits arise.

Can the reviewer also clarify in what way they believe the reproducibility is harmed by the lack of inclusion of the source code? The paper remains extensively detailed, giving every prompt used in Appendix D and the hardware, data sources, and split procedure in the Reproducibility Statement. Everything here can be reproduced quite easily using only standard tools, and we would argue the paper is vastly more reproducible in its current state than most contemporary work. We therefore fail to understand why the lack of inclusion of the specific source code at this moment remains a critical concern warranting rejection.

Reviewer_osK8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the feasibility of Predict-Answer/Act and Reason-Predict-Reason-Answer/Act paradigms where the model predicts how an external LLM judge would grade their responses. Authors propose two methods: one is providing an in-context "report card" containing the statistical mode of the model's historical grades, and another is applying supervised fine-tuning (SFT) on small models to directly output the ground-truth judge label, where the predictions are shown to be effective for a naive binary routing.

Critical strengths
  • The pre-hoc capability estimation and query routing are pratical problems and worth considering.
  • The paper evaluates a broad range of models across five benchmarks, providing a comprehensive results.
Critical weaknesses
  • The proposed contributions are very preliminary and lacks novelty/significance. The prompting mechanism with hand-crafted prompt template lacks justification and comparisons. The SFT approach is simply a 3-class classification problem where the training outcomes/dynamics are not studied thoroughly.
  • The shortcut learning problem is not addressed / discussed in prompting and SFT, where the model could just learn the query domain instead of the query prediction capability.
  • The presentation and flow of this paper are poor.
Questions for authors
  • Given that the mode scores for L321B and M09B are almost universally "bad" (Table 4), how do you distinguish genuine instance-level "self-awareness" from the model simply learning dataset-level domain classification or majority-class label bias?
2 comments
AuthorsOpenReview ↗

Response (1 of 2)

We thank the reviewer for their time and thoughtful comments.

W1 — "The proposed contributions are very preliminary and lacks novelty/significance. The prompting mechanism with hand-crafted prompt template lacks justification and comparisons. The SFT approach is simply a 3-class classification problem where the training outcomes/dynamics are not studied thoroughly."

We agree that the method is simple, but we believe this is a strength. The claim of the paper is not an algorithmic one. It is that a model can be made to predict, before answering, the grade an external judge would give its answer, and that this is enough to build a useful router from. The two ingredients (a report card that is nothing more than the model's modal grade per dataset, and a plain SFT run using the hindsight trick) are deliberately the simplest things that make sense here, and we consider them more valuable for it. We accept that the paper does not currently make this case well, and we will upload an updated copy of the paper before the end of the discussion period that does.

On justification and comparisons, the paper already contains three ablations that we did a poor job of emphasising. These are a different judge (GPT-OSS-120B in place of Llama-3.3-70B, Appendix I), a much shorter report-card template (where prediction accuracy drops noticeably due to weaker information, Appendix J), and a deliberately reversed grading rubric (Appendix K). We will emphasise this better in the updated copy of the paper.

On the SFT training dynamics, we do not study them, and we would argue they are not the object of this work. From the perspective of this work, the SFT is a means to an end and not part of the contribution in itself. Our analysis of it is limited to whether it is sufficient to greatly improve the predictive ability of the models (which it is) and whether it learns to predict the dataset or the properties of the query (which, as we argue under W2 and Q1, are largely the same thing here).

W2 — "The shortcut learning problem is not addressed / discussed in prompting and SFT, where the model could just learn the query domain instead of the query prediction capability."

This is the most important point raised across all three reviews, so we want to be precise about what we claim.

First, the goal of the paper is to predict the judge's grade. For the routing use case it genuinely does not matter whether a model arrives at the right grade by recognising the domain or by reading the individual question. Furthermore, since the gap between large and small models is usually concentrated in particular domains (e.g., maths), it should generally be sufficient to give a model an understanding of its abilities on different datasets. The report card is in fact designed to inject exactly that domain-level prior. So we do not regard domain recognition as a shortcut in the pejorative sense. It is most of what the method is supposed to supply.

Second, the question of whether anything beyond the domain prior is being learned is still a fair one, and we have now measured it directly. For every dataset and fine-tuned predictor we computed the three-way accuracy with a 95% Wilson interval and compared it to the majority baseline (always predicting that model's most common grade on that dataset). Any cell whose interval lies entirely above the baseline is doing something that domain recognition alone cannot do, because domain recognition caps out at the baseline.

DatasetFine-tuned predictorAccuracyMajority baselineSkill
MMLU-ProLlama-3.1-8B0.5540.374+0.180
MMLU-ProLlama-3.2-3B0.5690.487+0.082
LongFactLlama-3.2-1B0.4620.388+0.074

These are the cells whose 95% interval excludes the majority baseline. In the updated copy of the paper we will upload before the end of the discussion period, we will report this analysis in place of the current per-dataset figures, and drop any wording that implies instance-level insight where the interval does not support it.

Third, the paper's CommonsenseQA hold-out (Appendix H) is a dataset that was never used to build a report card or to train a predictor, and the predictors that gained the most from our methods keep most of that gain there. That rules out the pure dataset-memorisation version of the shortcut.

AuthorsOpenReview ↗

Response (2 of 2)

Q1 — "Given that the mode scores for L321B and M09B are almost universally 'bad' (Table 4), how do you distinguish genuine instance-level 'self-awareness' from the model simply learning dataset-level domain classification or majority-class label bias?"

The table in W2 is our answer to the "how" part. Where a predictor beats the majority baseline by a margin whose interval excludes it, the gain cannot be domain classification or label bias, since both of those stop at the baseline. That happens for Llama-3.1-8B and Llama-3.2-3B on MMLU-Pro and for Llama-3.2-1B on LongFact. For MobileLLM-0.9B, and for Llama-3.2-1B on the datasets where its grade is "bad" almost everywhere, we cannot make the distinction and we will stop implying otherwise.

Beyond that we deliberately do not try to separate the two, because we do not think they are separate things. Much of what a model can know about how it will do on a query comes from the properties of the query (its domain, its difficulty, its format), and most of that is captured by which dataset the query comes from. For the small models in question, a predictor that has learned how the model does on that dataset has learned most of what there is to learn about the query. The report card is built to supply exactly that, and for the routing use case it is sufficient.

We also take the point about the word "self-awareness". The quantity we measure is agreement with an external judge's grade, and in the updated copy of the paper we will upload before the end of the discussion period, we will describe it more precisely.

W3 — "The presentation and flow of this paper are poor."

We partly agree, and we will not try to argue this. In the updated copy of the paper we will upload before the end of the discussion period, we will (i) cut Section 2 to a short preliminaries block, removing the transformer notation that is never used again, and (ii) replace the five similar per-dataset figures with one compact presentation. We would be grateful for any further specific pointers the reviewer has on where the flow breaks down.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Update of the Paper

We have updated the paper with the changes promised several days ago. We hope that the reviewers have had a chance to review our responses to their points. Due to the time needed to code and run the additional experiments that were requested by the reviewers, it was not logistically feasible to provide a response earlier and we thank the reviewers for being understanding. At present, the main text now sits at 10 pages with 42 pages of appendices and we hope this is sufficient to address all concerns. As such, we would also like to bring to the reviewers' attention the impending discussion period deadline.

For ease of reference, the changes are as follows:

  • Section 2 is cut to a short preliminaries block (osK8 W3, nY5q W1, zViV W3).
  • The case that the simplicity of the method is a strength is now made, and the ablations in Appendices I, J, and K are emphasised (osK8 W1).
  • The Wilson interval analysis is reported in Section 5.1, and wording implying instance-level insight is dropped where the interval does not support it (osK8 W2).
  • The quantity we measure is described more precisely (osK8 Q1).
  • The judge-validity check is added as Section 3.2, alongside the self-consistency check (nY5q W2).
  • The constant-"bad" distinction is made explicit, and the token costs are stated alongside the routing table (nY5q W3, zViV Q3).
  • The contextual summary table is moved to the main text as Table 1 (nY5q W4).
  • RouteLLM, Meta-Router, and R2-Router are added to the related work (zViV W1b).
  • The joint-grading limitation is stated in Appendix C (zViV W2b).
  • The split reasoning is made explicit in the reproducibility statement (zViV Q1).
  • The newer-model, within-domain transfer, and MATH/BigCodeBench experiments are added as Appendices P, O, and N (zViV W1a, W2a, Q1).

Appendix letters are unchanged; table numbers have shifted.

Two deviations were made between the changes we stated we would make and the revised paper. First, we experimented with a condensed version of Figures 7 through 11, but found that any such condensation was quite destructive to the data presented there, as averaging over datasets hides the diversity of the contrasting performance on different datasets, which we feel unfairly favours our results over baseline ones. The contextual summary table now in the main text as Table 1 helps here, and we would be happy to take suggestions for a modification to these figures for a camera ready version of the paper. Second, due to non-trivial anonymity concerns, we are holding off on including a link to a self-contained, anonymised version of the code. This is due to the fact that disentangling it from potentially identifying sources would greatly limit the utility of the code without large-scale rewrites. Since it would be quite unusual for a reviewer to run the code themselves in the review process (at quite a great expense to themselves), we assume the reviewers would prefer us to avoid any risk of breaking anonymity here over releasing a version of the code immediately. The code is slated for a public release in a camera-ready version of the paper as noted in the reproducibility statement.

AuthorsOpenReview ↗

End of discussion period

As the discussion period draws to a close, we would like to thank the reviewers and the Area Chair for their time. We have responded to each point raised and updated the paper accordingly. We appreciate that this year's author-reviewer discussion period is shorter than usual, and we hope it has been enough for Reviewers osK8 and nY5q to read our responses. As no further questions have come in, we hope this means their concerns have been addressed. We have also replied above to the follow-up from Reviewer zViV, and we hope this further helps to address their remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.