acceptodds
Under review as a conference paper at ICLR 2027

SELECTION REGRET: LOCAL ACCURACY MISRANKS CLOSED-LOOP RELIABILITY

Abstract

Local next-transition accuracy is a tempting way to choose components for a system that will eventually run in a loop. This paper shows that the metric can pick the wrong component once that system is evaluated end to end. In a controlled frozen-state routing setup that holds state access and termination fixed while varying transition scorers, teacher-forced edge accuracy is nearly identical across scorers—about 50–53%—but exact constrained-route accuracy differs by 36–88 points. Across 1,800 scorer-choice contexts, choosing by teacher-forced edge accuracy matches the closed-loop winner only 62.4% of the time. Mean selection regret is 9.84 points, compared with 13.37 for uniform-random selection; the 95thpercentile regret is 86.56 points, and 40.7% of contexts have tied top local scores. The mismatch persists when rank correlation is high: at L=4, correlation is 0.874 while selection agreement is 53.6%. Gold-prefix controls show that some failures precede generated-prefix feedback. Matched length-four continuation supervision brings L=4 accuracy from near zero to 75–82%; STOP-only data do not help, and margin-guided selection does not outperform random selection. On 90 AppWorld tasks and five backbones, a route-only metric selects a model with 0% task success under one official scaffold and 7.8% under a second; the best model reaches 13.3% and 23.3%, respectively. An offline metric that also checks call arguments selects that best model in both settings. Thus local next-step accuracy alone does not reliably select a component for closed-loop use. On the controlled surface, tied top local scores account for 64.2% of selection regret and can be detected without rollout.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model selection,Closed-loop evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_w2NjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a component that scores well under teacher forcing is also a good choice when the system runs in a loop. The authors compare transition scorers using the same frozen representations and state-access mechanism, then measure the regret of selecting a scorer by its local accuracy. Gold-prefix and continuation-training experiments help locate the failures. On AppWorld, a route-only metric selects a poor backbone under two agent scaffolds; including call arguments recovers the best aggregate backbone on the evaluated tasks. The paper's main contribution is the evaluation study and its controls.

Critical strengths

The controls are the strongest part of the paper. State access and termination are measured separately, and the gold-prefix experiments help distinguish a bad conditional transition from errors caused by earlier generated actions. The continuation versus STOP-only comparison is also useful: it shows what additional supervision repairs, and at which horizon. The negative results for wider beam search and margin-guided example selection help keep the conclusions grounded (Section 5 and Appendices J-L).

Selection regret is a sensible quantity to report here. A high rank correlation can look reassuring while still giving the wrong deployment choice, and the paper makes that distinction concrete. AppWorld adds a real end-to-end example. I also appreciate that the authors separate this experiment from the frozen-prefix checks and state the limits of the same-task comparison.

Critical weaknesses

My main difficulty is understanding exactly what is being compared in Table 1. For Gemma/Scorer B, teacher-edge accuracy is 51.3%, while exact constrained-route accuracy is 97.5%. If both metrics used the same routes and admissible candidates, a correctly decoded route would also give correct gold-prefix decisions along that route. The caption says the teacher metric includes STOP and comes from an all-valid-edge evaluator, but I could not reconstruct its numerator, denominator, or evaluated edge population from the paper. There may be a straightforward explanation. Until it is clear, though, I am unsure how much of the headline gap comes from a weak local predictor and how much comes from different evaluation definitions.

The AppWorld result also needs a clearer distinction between choosing one backbone for the whole pool and choosing separately for each task. Appendix M reports tied top full-call scores on 80 of 90 tasks, yet the full-call metric selects Qwen3-30B on 88 tasks. With so many ties, the per-task result could depend heavily on the tie-breaking rule. The aggregate ranking is still useful, but the reported selection frequency and regret are harder to interpret. The authors already acknowledge that selection on unseen tasks has not been tested; I would keep the practical conclusion within that scope.

Questions for authors
  1. How is teacher-edge accuracy in Table 1 computed? Please specify the evaluated edges, candidate masks, STOP handling, and averaging. What is the gold-prefix skill accuracy on exactly the routes and candidates used for constrained decoding?

  2. What happens to AppWorld full-call selection regret under random tie-breaking? How much of the 88/90 Qwen3-30B selection frequency comes from tied scores?

  3. Can you break down local scores and regret by the declared core/development/confirmation partitions? Selecting on one partition and evaluating the chosen backbone on another would help establish what the selection result supports.

Reviewer_f3pZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks a simple question: if you pick a model by how well it predicts the next step on a test set, do you also get the model that works best when it runs on its own? On a test set, the model always sees the correct earlier steps. When it runs on its own, it sees its own earlier answers, so one mistake can lead to more. The paper calls the loss from picking the wrong model this way "selection regret". The authors test this in three settings. (1) A toy task: apply a short sequence of simple text edits (e.g., move the first letter to the end, replace vowels with #). Four small models of the same size, built on top of three fixed large language models (LLMs), predict which edit comes next. (2) Two small checks: a toy chain of tool calls, and SCAN, a standard benchmark that turns short commands like "jump twice" into action sequences. (3) A real agent benchmark, AppWorld. There they compare two ways to grade the next step: checking only which tool is called, or also checking the inputs given to that tool.

Critical strengths
  1. The idea is simple and clear because it asks a practical question that anyone building an agent faces: does the model that looks best on a step-by-step test also work best when it runs on its own?
Critical weaknesses
  1. found two wrong citations looking like ai generated list below
  • A Benchmark for Systematic Generalization in Grounded Language Understanding <- author name wrong
  • Neural Module Networks for Reasoning over Text <- author name wrong
  1. The main lesson is already known. A model checked with the correct earlier steps can fail once it must use its own earlier answers. This is the well-known exposure-bias problem, and the paper itself cites the standard fixes (scheduled sampling and DAgger).

  2. The toy-task results are not stable, so I do not trust the main gap. Across the paper's tables, the same small model (Scorer A) gets anywhere from about 0% to about 100% on length-3 programs (Tables 1, 3, 15, 21), and almost every single run ends at either 0% or 100% (Table 17). So the large gaps between the four small models may come from lucky runs and settings, not from the models. In one rerun, Scorer A scores exactly 40.00% on all three LLMs, which makes me wonder whether it uses them at all.

  3. The offline score at the center of the paper is not clearly defined ("the artifact's all-valid-edge metric"). It counts the stop decision, but in the tested system the length comes from a separate part.

  4. The writing is hard to follow. The main text is full of seed numbers and protocol labels ("locked", "errata ledger"), the four models are hidden as Scorer A-D, and terms like "consumed interface" are never explained.

Questions for authors
  1. Can you re-run Metric B for Gemma with a prompt it does not refuse, or score refusals separately? Does Metric B still pick Qwen3-30B?
  2. How many AppWorld tasks would be needed to separate close models (Qwen3-30B 13.3% vs Qwen3-8B 10.0%)?
  3. Beyond exposure bias and scheduled sampling, what should a practitioner learn here that they would not already know?
Reviewer_oH2cOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper primarily presents diagnostic experiments examining whether local evaluation metrics can reliably guide model selection for multi-step sequential tasks. The work does not present a new offline evaluation methodology, in fact the experiments largely reinforce the issues with local metrics. However, the AppWorld experiments do provide a useful observation that evaluating both tool choice and call arguments improves model selection.

Critical strengths
  • The paper presents a set of experiments to examine the reliability of local metrics, the breadth of the experiments is definitely something worth noting.
  • The paper gives some reasonable insights on the failure of local metrics.
Critical weaknesses
  • Overall paper was very hard to read. The paper starts off with a lot of numbers on the abstract itself and it smells of heavy LLM usage.
  • I do not think the paper merits an ICLR acceptance, the paper does not present any new offline evaluation methodology and most presents a bunch of diagnostic experiments to show why local metrics are not fully reliable, the paper lacks novelty on that front.
Questions for authors

Nothing major. I think the paper needs a lot of work in its present form - both in articulation and experimental novelty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.