acceptodds
Under review as a conference paper at ICLR 2027

When to Stay, When to Switch: Outcome-Based Evaluation of Adaptive LLM Inference

Abstract

Adaptive LLM inference chooses a model or reasoning budget for each request. Evaluating these choices after an input change requires knowing whether the chosen computation still produces a good answer at an appropriate cost. A route can stay unchanged while its outcome becomes inadequate; a switch can be necessary for a good outcome. We introduce PO-CTRLBench, a paired evaluation framework that measures every available action's quality and cost through repeated execution. These outcome profiles distinguish three requirements: keep an adequate action, allow a change among adequate actions, or adapt when no action remains adequate at both inputs. We show why routes alone cannot identify these requirements, separate outcome failures from selection inconsistency, and give finite-sample certification rules. Our controller, PORC, learns the profiles and selects actions using calibrated conservative utility scores. On two-action reasoning tasks, route consistency and endpoint risk rank five methods in exactly opposite orders. Compared with Outcome-greedy, PORC reduces the primary endpoint-risk sum from 12.1% to 2.2% and improves utility by 0.064 (95% source-cluster confidence interval [0.049,0.079]). Under source shift, changing the decision scores on the same predictor reduces full paired risk from 23.89% to 12.50% and raises utility by 0.040. Paired outcome evaluation thus makes the distinction between stable computation and appropriate computation measurable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, adaptive inference, model routing, test-time compute, outcome-based evaluation, robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MfmdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark to evaluate whether or not an LLM is good at choosing its model and computation budget. They do this by considering prompt pairs and feeding them separately into the LLM. They then observe the computation budget and model that the LLM chooses in each case. The LLM either uses the same setting, or switches. The question is whether this decision was justified or not. To answer this, the authors propose forcing the chosen setup and its alternative to the LLM and analyzing its reponse to the prompt pair many times. This allows them to experimentally observe which option gives them the best trade off between cost and action quality. Based on this, they can know whether or not the original choice of the LLM was justified.

Additionally, the authors introduce a method for choosing the model and computation option. This is done by training a NN on the offline data used for the benchmark to learn the quality-cost trade-off given the setting, and then choosing the best one.

The authors test PO-CTRLBench and PORC on a number of experiments, and compare them to some benchmarks. PO-CTRLBench shows that route-based metrics for choosing a model might be misleading and not choose the option that has the best quality-cost tradeoff, while PORC is seen to decrease token cost while also increasing quality compared to existing benchmarks.

Critical strengths

I think the paper's claim about how adaptive inference systems are currently evaluated is very interesting and well discussed. Essentially, only looking at consistency of the route is not an enough metric, and the authors do a good job justifying this both in the text and via the experiments.

Critical weaknesses

1- One of the main lacks of the paper is not clearly mentioning the actual consistency rates in the experiments. The authors only mention the ranks of the methods obtained using route concsitency, but not the actual rates. This makes the claims difficult to verify.

2- I have a concern regarding how cost is compared in the experiment with three models (0.5B, 1.5B, and 7B). It seems like the authors use the number of output tokens as the main measure of cost. However, producing a token using the 7B model, for example, is going to be much more expensive than the 0.5B model. This way of measuring cost ignores many aspects such as model size, latency, and computational cost.

3- It seems like the main comparison of the five different routing protocols only uses the PPFR and CRMR metrics, which to my understanding, only calculate the quality and correctness of the edited prompt.

Questions for authors

1- Can the authors report the actual consistency rates for the experiments? (regarding W1) 2- Can the authors discuss how cost can be calculated well in situations like those explained in W2? It would be beneficial for the paper if the authors could re-evaluate the result using a more realistic cost measure. 3- It would be great if table 3 could also report metrics that report the quality of the answer to the original prompt as well, for the sake of completeness of the reported results (Regarding W3).

Reviewer_xhmNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PO-CTRLBench, which measures the quality and cost of each action before input editing and the changes after editing to define three types of relationships: Equivalent, Preserving, and Critical. It distinguishes between endpoint actions that are unreasonable and those with inconsistent selections, and provides risk decomposition and finite sample verification rules. The authors further propose PORC, which combines action result prediction, paired training, and conservative utility score to select actions. In a dual-action task, PORC reduces the sum of endpoint risk components after editing from 12.1% to 2.2%, while improving the average utility; the source change experiment also shows that it can reduce the complete pairing risk. The main contribution of this paper is to establish an adaptive computing evaluation framework based on results, but the correspondence between statistical verification and experimental labels, as well as the validation of the main experiment on the complete framework, are still insufficient.

Critical strengths
  1. The paper points out that maintaining the original action after input changes is not necessarily reasonable, and switching actions does not necessarily indicate a lack of robustness. By shifting the evaluation focus to the actual quality and cost brought about by the actions, the paper raises a practical issue: adaptive reasoning should determine when to maintain and when to adapt, rather than merely pursuing a lower switching rate.
  2. The proposed PO-CTRLBench repeats all possible actions on both the original and edited inputs. It distinguishes equivalent, maintainability, and necessary adaptability through the near-optimal action set. This design avoids predefining correct behavior solely based on the editing type and separates unreasonable endpoint actions from inconsistent additional choices, making the evaluation results more interpretable.
  3. The designed PORC explicitly predicts the quality and cost of each action, combines paired supervision with calibrated residual adjustment utility scores, and the method is lightweight and has clear decision logic. The paper also provides risk decomposition, finite-sample verification rules, and conditional regret value bounds, providing a clear mathematical basis for evaluation indicators and conservative choices.
  4. The experiment covered dual-action reasoning, source distribution variation, and three-action model selection, and isolated the influence of score adjustment through the PORC-point/PORC-LCB shared predictor. The source-level data partitioning and clustering bootstrap retained the correlations among editing, checkpoints, and repeated executions; the fixed action baseline and unverified record analysis also helped to present the trade-offs between average utility, selection risk, and computational cost.
Critical weaknesses
  1. The main experiment executed each action and each prompt 128 times. In the case of dual actions, a total error budget of 0.05 was allocated to 12 intervals, and the result summary was certified as equivalent with a cost difference tolerance of 0.05. However, according to the Bentkus reference inversion and signed difference scaling rules in Appendix A.2, even if all 128 paired differences were zero, the resulting difference interval would be approximately [−0.282,0.282][-0.282, 0.282], which cannot be certified to fall within [−0.05,0.05][-0.05, 0.05].
  2. The complete framework employs the E/P/C relationship, requiring reasonable actions at both ends and adding consistency requirements to the E category. However, the main experiment adopts the Eprof/C/UE_{\mathrm{prof}}/C/U protocol, only counting endpoint failures after editing, and uses γcert=0.03\gamma_{\mathrm{cert}}=0.03 for authentication and γeval=0.05\gamma_{\mathrm{eval}}=0.05 for evaluation. Therefore, the 12.1% to 2.2% mentioned in the summary cannot be interpreted as a decrease in the risk of complete pairing.
  3. For the two actions, PORC-LCB is equivalent to shifting the decision threshold of the utility difference from zero to eT−eSe_T - e_S, and this shift is fixed within the calibration group. Therefore, the current results prove that a certain fraction offset is effective, but it has not yet been proven that residual calibration is superior to directly optimizing the threshold selection for utility or risk constraints on the validation set. Furthermore, the main experiment PORC, along with Outcome-greedy, simultaneously changes the paired training objectives and decision rules, making it impossible to separate the contributions of the two.
  4. The main experiment had only 60 independent test source questions, while the three-action experiment had only 32. The extensive enforcement has improved the measurement accuracy of the individual result overview, but it did not expand the diversity of the questions. The combination of the TF-IDF predictor with the limited editing template also requires the exclusion of template dependence through tests with unseen editing types.
  5. The three-action experiment charges only based on the number of generated tokens for different models, without considering the computational load, latency, or monetary cost corresponding to the model size. Therefore, currently supported is the selection improvement under a specific token cost definition, and it is not yet able to fully support the conclusion of real cross-model computing efficiency.
Questions for authors

My concerns are illustrated as weaknesses.

Reviewer_qP5nOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a paired evaluation framework (PO-CTRLBench) to evaluate adaptive LLM routers under input edits. In this framework, the quality and cost of all candidate actions are measured before and after an edit, which determines when a router should stay with the same action or switch to another action. Additionally, the conservative utility controller (PORC) is proposed as a routing method.

Critical strengths

The evaluation framework is clear. Decomposing paired risk into endpoint inadequacy and equivalent-pair selection inconsistency (Proposition 1, Rfull=RB+SER_{\mathrm{full}} = R_B + S_E) provides a useful perspective on when a router should stay or switch.

The controlled predictor comparison is reasonable. Holding the outcome predictor fixed when comparing point scores and conservative scores in Table 4 is a clean design to test the decision rule.

Critical weaknesses

Lack of necessary references. The Introduction only cites two references when discussing the core problem of router stability under prompt edits, failing to properly elaborate on the background knowledge and related literature.

Poor presentation and lack of illustration. This paper is crudely prepared, full of dense text and equations, and lacks any clear framework diagram or visual illustration, making it very hard to read and understand. Furthermore, the overall presentation is messy and disorganized, reading very much like an AI-generated paper.

Limited novelty and insufficient experimental results. The technical novelty is far from sufficient to support an ICLR paper. Specifically, the evaluation framework merely applies simple threshold rules to empirical quality–cost utilities, while the proposed PORC controller simply feeds shallow TF-IDF + SVD features into a 128-unit single-layer MLP with a naive conformal residual offset. In addition, the reported experimental results are too sparse, covering very few settings and tiny test splits (e.g., only 64 pairs in the three-model routing experiment).

Unreproducible certification results. The 396 primary EprofE_{\mathrm{prof}} labels cannot be derived from the interval formula in Appendix A.2. Under Eqs. 17–19 with m=128m=128 and δ=0.05/12\delta=0.05/12, even a zero mean difference produces a confidence interval of [−0.282,0.282][-0.282, 0.282], which is wider than both the quality tolerance [−0.15,0.15][-0.15, 0.15] and cost tolerance [−0.05,0.05][-0.05, 0.05]. Thus, the stated formula yields 0 certified EprofE_{\mathrm{prof}} pairs rather than 396.

Missing threshold baseline. For two actions, Eq. 14 simply applies a constant threshold eT−eSe_T - e_S to the predicted utility difference U^T(x)−U^S(x)\widehat{U}_T(x) - \widehat{U}_S(x). The paper does not compare PORC against directly tuning a scalar threshold on the validation set using the same predictor.

Questions for authors

Please refer to the weaknesses part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.