acceptodds
Under review as a conference paper at ICLR 2027

Selection Is Not Evaluation: Maximization Bias in Scoring Runtime Interventions for Language Agents

Abstract

Runtime interventions for language agents, such as a warning, a new plan, or a rollback, are often judged by branching from a shared prefix and crediting the intervention whose rollout succeeds. Because rollouts are stochastic, this credits the best of several noisy outcomes, the same maximization bias that motivated Double Q-learning. We propose a repeated-branch protocol that gives every action, including no intervention, two independent rollouts from the same state, selects the action on one and scores it on the other, and fixes all controller predictions before any outcome is observed. On held-out ALFWorld tasks with a Qwen3-8B actor, scoring on the same rollout overstates the value of intervening by 15.4 percentage points. With this bias removed, a controller trained to predict each intervention's benefit over continuing gains less than two points over never intervening. That difference is not significant, and the controller does no better than one that predicts per-action outcomes. On ScienceWorld the gap is smaller, and no controller improves on continuing. A rescue seen on a single rollout is therefore weak evidence that intervention helps. Evaluations should include continuing as an action and score each choice on a rollout that was not used to make it.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, runtime intervention, agent evaluation, maximization bias, winner's curse, policy evaluation, stochastic rollouts, ALFWorld

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Sz7KOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is about doing runtime intervention for LLM agents, ie we put an LLM agent (Qwen3-8B) into a multi-step env (ALFWorld, with ScienceWorld as a small boundary panel), stop at one point in the trajectory and predict what to insert: continue, warn, replan, rollback (and in the second study also a three-step rollback and a “name what you still need and where you haven’t looked” message). From the same saved state, they run every action twice with independent seeds, so an action can be chosen on one draw and scored on the other, and they show that the usual choose-and-score-on-the-same-branch evaluation overstates value by 15.36 points on 384 tasks, which doesn’t exceed what you get by shuffling action labels within each task (17.26 on eligible prefixes), so it’s basically continuation noise.

On top of these branches the author also fit random forests on some hand engineered features (counts of invalid actions, reward summaries, TF-IDF of recent text etc) to predict either each arm’s outcome or its benefit over continuing, plus a “cross-draw” version that picks the arm with one draw’s forest and values it with the other’s, and all predictions are frozen before any arm outcome exists. In the first study (decision at a fixed early step) no controller beats continuation significantly and none beats always-replan, and in the second (decision at the first invalid action or unchanged observation, 593 tasks) the cross-draw controller gets 56.24% vs 54.97% for continue and 54.30% for the best fixed repair, but neither gap passes their pre-registered test.

Critical strengths

None

Critical weaknesses
  1. The setting itself is sound, however the execution and experiment setup is less thought out. For example, why do we need a “warn” in a selection choice when no obvious context or signal requires it? Also the study is conducted with only single LLM in a single experiment environment + limited settings environment. I personally highly doubt the observations found so far to be even generalizable to broad LLM behavior or whether it's just because the model is too small, suppose we scale the model size does behave better? What about LLMs equipped with agentic behavior such as Ornith 9B?

  2. Based on the experiment results, I’m not convinced that the learned controller falls to select “always repair”, which is obvious in hindsight. If I have to blindly guess from 4 interrupt keyphrases : continue, warn, replan, rollback, of course I would choose replan, same like S1 [arxiv:2501.19393] where you just add the same keyphrase (“Wait”) to get a long CoT.

Questions for authors
  1. Why is a random forest the initial choice and not a linear model? It seems odd that this is the case, as far as I can see there’s no linear or logistic model anywhere in the paper and citing Breiman is the only justification. The only alternatives are on p.16 (gradient boosting, 0.6B LM embeddings, and both together), at 29.27/29.10/28.84 vs 29.47 at the scheduled point and 49.46/50.29/50.15 vs 50.69 at the triggered one. These are all within about a point, so I’m not sure I can read that as “model class doesn’t matter”, it may just be within dev noise (also, why is scheduled dev utility 29.47% when confirmation success is around 55%?).

  2. The main text says 593 tasks run under both rules (497 and 540 eligible), but Appendix D and Table 11 say both rules run the same 288 identities, and 593 can’t be balanced over six task types either. Which is right?

  3. One missing discussion which is what is the connect of this setup to contextual bandits? Since contextual bandits is basically : “pick one of K actions given context” but the paper never compares to or discusses them. I get that with every arm run twice on every dev prefix the data is full information rather than bandit feedback, but that makes a simple per-arm model even easier to fit, it pools across tasks the same way your Efron & Morris argument wants, and it’s the natural thing if this were deployed on a stream of incoming tasks where you only see the arm you fired.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Sz7K

We’re sorry for the contradiction you found in the task counts. 593 is the correct number under each checkpoint rule. The first 288 identities were balanced, with 48 in each of six task types. We then added 305 identities from the remaining inventory. That combined set was unbalanced. Of the 593 tasks, 497 reached an event checkpoint and 540 reached a scheduled checkpoint, with 487 eligible under both rules. We checked the identities against the recorded runs and corrected the main text, Appendix D, and the table captions.

Your fixed-prompt comparison was well chosen. The warning asks the actor to check its goal, latest observation, and previous action. At a scheduled step it can interrupt an agent that is doing fine. Always replanning was therefore an important baseline, and on Study 1 ALFWorld it scored 55.34%, ahead of the learned direct forest at 54.43% and continuing at 52.86%. The learned rule did not collapse to replanning. It continued on 245 of 355 eligible prefixes, replanned on 106, and warned on four. Study 2 gave a different answer. At failure signals the controller continued on 298 prefixes, warned on 36, and asked the actor to locate missing objects on 163. It never chose replanning there. Section 5 now reports what the controllers actually did.

We also ran the per-action regression you suggested. Your contextual-bandit analogy describes the execution decision well. During training, restoring each development prefix gave us two observed outcomes for every available action, so we could fit one logistic model per action directly. On held-out Study 1 ALFWorld tasks, logistic regression scored 53.91% versus 54.04% for the outcome forest. The paired difference was −0.13 points, with a 95% interval from −2.66 to 2.41. Ridge regression on gains over continuing scored 54.30% versus 54.43% for the direct forest. We agree that the small differences among boosting, embeddings, and forests on development data could not settle the question of model capacity. Appendix J now reports the held-out linear comparisons and their uncertainty.

On actor size, the planned studies used Qwen3-8B. We do have a separate 27B experiment, but it also changed the policy, filtering, and decoding. It cannot tell us how the same intervention controller would behave if only actor size changed. We have made that distinction explicit in Appendix G.

We also ran the per-action regression you suggested as LR

However, given that logistic regression and random forests achieve almost identical performance (53.91% vs. 54.04%), I think a more fundamental question remains: what is the learned controller actually learning, and why should we expect it to work?

Does this similarity suggest that the intervention decision depends on relatively simple features, or that the available features contain too little signal to reliably distinguish between interventions? In particular, it would be helpful to understand whether the controller learns meaningful context-dependent intervention choices or primarily reproduces a preference for certain actions.

Since neither learned controller demonstrates a statistically significant improvement over the fixed baselines, I still find it difficult to distinguish genuine intervention-selection ability from noise. The additional regression baselines clarify that model complexity is probably not the main issue, but they do not yet explain whether the proposed learning approach captures useful intervention signals.

The learned rule did not collapse to replanning. It continued on 245 of 355 eligible prefixes, replanned on 106, and warned on four.

While the action distribution is useful, it does not necessarily demonstrate that the controller has learned when to intervene. Could the authors provide evidence that the selected interventions outperform alternative actions on the particular states where they are chosen? Since all intervention outcomes are available in the experimental setup, a conditional analysis of intervention gains would help distinguish meaningful context-dependent selection from simply learning an overall action preference.

separate 27B experiment, but it also changed the policy, filtering, and decoding. It cannot tell us how the same intervention controller would behave if only actor size changed.

However, this leaves my original concern about generalizability largely unresolved. In particular, how much of the observed difficulty in selecting effective interventions is attributable to the controller, and how much is specific to the Qwen3-8B actor? Without a controlled comparison across actor capabilities, it seems difficult to establish whether the reported conclusions extend beyond this particular agent configuration.

AuthorsOpenReview ↗

Does the controller choose the right repair for the state?

Your question points to a comparison we should have reported. The logistic model and the outcome forest chose the same action on only 142 of 355 eligible ALFWorld states (40%), despite scoring 53.91% and 54.04% over all planned tasks. Similar success rates therefore tell us little about which rule either model learned. To test whether the choices were useful, we went back to the two held-out outcomes for every action. The controllers’ choices had been recorded before those outcomes were generated.

Where did the chosen repair help? In Study 1, on the 106 states where the direct forest requested a new plan, replanning succeeded 61.32% of the time and continuing succeeded 55.66%. But replanning also beat continuing by 1.43 points on the 245 states where the forest chose to continue. The difference between those gains was 4.23 points (95% interval −2.58 to 11.13). On this panel, we cannot distinguish reliable targeting from an overall preference for replanning.

Study 2 gives a more informative result. At a failure signal, the controller sometimes asked the agent to name the objects it still needed and the places it had not searched. On the 163 states where it chose that reminder, the reminder scored 45.09% versus 40.49% for continuing. On the 298 states where it chose to continue, the same comparison went the other way: 49.16% versus 50.84%. The reminder’s relative gain differed by 6.28 points between those groups (95% interval 0.30 to 12.32). The remaining 36 states received a warning.

We also tested your action-preference explanation directly. We kept the exact frequencies of all chosen actions and calculated what success would have been, on average, if those same choices had been assigned at random across eligible states. At failure signals, the frozen assignments scored 50.40% versus 48.64% for that baseline, a gain of 1.76 points (0.34 to 3.17). Holding the mix fixed within each of the six task types still gave +1.61 points (0.21 to 2.94). Holding it fixed within each floorplan gave +0.70 points (−0.66 to 2.04), so the data do not resolve how much the controller picks up within a layout. At scheduled checkpoints the same frozen controller gained −0.07 points over the randomly assigned mix (−1.62 to 1.47). These are retrospective comparisons with pointwise 95% intervals from 50,000 whole-floorplan bootstrap resamples. They use eligible states only. The planned comparison against always continuing still includes zero.

Our reading is that the failure-triggered controller picked up some information about where the reminder was useful beyond how often it preferred that action. The data do not identify which prefix feature carried that information. On your actor-size question, we agree that the 27B experiment cannot separate actor capability from controller quality. We would need the same checkpoints, repairs and budget with both actors, testing the frozen controller on each and then refitting it for each actor. What transfers without that experiment is the evaluation procedure. On these data it distinguishes three empirically different things: a rollout that makes a repair look good in hindsight, a repair that tends to help on the states where the controller chooses it, and a controller that beats a fixed policy over whole episodes. We found evidence for the second at failure signals. The third remains open.

Reviewer_taq6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work addresses "same-draw optimism" in LLM agent runtime interventions, a maximization bias where evaluating an intervention on the same stochastic rollout used to select it inflates reported performance. To resolve this, the authors propose a repeated-branch protocol that generates two independent continuations (Draw 0 and Draw 1) from a shared prefix to decouple selection from evaluation. The paper also shifts interventions from fixed steps to event-triggered failure points and introduces a cross-draw random forest controller. Experiments across ALFWorld and ScienceWorld using a Qwen3-8B actor show that same-draw evaluation inflates success, but the proposed controller fails to achieve significant gains over baseline continuation.

Critical strengths
  1. The paper clearly highlights how evaluating runtime interventions on identical stochastic continuations introduces substantial optimism bias.
  2. The commitment to pre-registered prediction freezes, hashed feature specifications, and detailed compute logging sets a commendable standard for benchmark transparency.
  3. Demonstrating that event-triggered intervention points outperform fixed-step scheduled checkpoints offers a helpful conceptual insight regarding decision timing.
Critical weaknesses
  1. The core empirical claims are fundamentally unconvincing because the proposed controller yields statistically non-significant improvements over baseline continuation on both scheduled and event-triggered panels. The authors admit the evaluation panel was underpowered to certify these gains.
  2. The empirical evaluation is restricted to simplistic text environments (ALFWorld and ScienceWorld) with Qwen3-8B using ReAct prompting. Testing on real-world coding or web-agent tasks is completely absent.
  3. Generating multiple independent rollout branches per action requires massive inference overhead (over 250 million input tokens), which is practically unviable for a meager, non-significant ~1.26 percentage point gain.
  4. The learned controller relies on classical random forests fitted on crude numeric features and SVD-reduced TF-IDF text features. This shallow machine learning stack feels severely dated for managing complex LLM agent dynamics.
  5. The paper obscures simple, well-known statistical concepts (Double Q-learning and holdout validation) under pretentious terminology like "same-draw optimism", "reading the room", and "signed-benefit forests".
Questions for authors
  1. Why did you choose shallow Random Forests with TF-IDF features rather than training a neural reward model or fine-tuning an LLM controller?
  2. Given that the primary contrast against continuation is statistically insignificant, why do you claim deployable utility rather than concluding that the intervention is ineffective?
  3. How can this repeated-branch protocol scale to real-world agent environments (e.g., web browser automation or live software engineering) where state serialization and deterministic resets are impossible?
  4. Why did ScienceWorld perform so poorly across all policies without producing any meaningful controller differentiation?
  5. Did you evaluate under greedy decoding (temperature = 0), where stochastic continuation noise—and thus same-draw optimism—trivially disappears?
1 comment
AuthorsOpenReview ↗

Response to Reviewer taq6

We’re sorry the original abstract gave the impression that we had established a deployment benefit. We agree that the primary comparison cannot support that conclusion. On 593 ALFWorld tasks, the event-triggered controller reached 56.24% success versus 54.97% for continuing, a paired difference of 1.26 percentage points with a 95% interval from −0.35 to 2.83. The data allow both a small loss and a modest gain. We have revised the abstract and conclusion accordingly. We also corrected the interpretation of timing. Both rules used the same task identities, but they generated separate trajectories and reached different eligible states. Their results do not establish that waiting for a failure signal is generally better.

Why a forest? We wanted to compare two training targets while holding the features, development tasks, and learner capacity fixed. Study 1 had 129 eligible ALFWorld development tasks. We have now fit one logistic regression per action on those same data. It reached 53.91% on held-out tasks, compared with 54.04% for the corresponding forest. This answers a narrower question than a neural reward model or an LLM supervisor would answer. Section 6 specifies the common prefix, action choices, and remaining budget needed for that comparison.

What did the 250 million tokens buy? The 253.76 million known input tokens covered the first study’s offline evaluation, including baseline episodes, alternative branches, and retained attempts. A frozen controller executes one chosen continuation. If the only question is whether that controller beats continuing, comparing those two policies directly is cheaper. The additional branches let us rescore the same hindsight choices and measure how much the original evaluation inflated success. This requires a reproducible checkpoint. An isolated browser backend or software test fixture could provide one. For an irreversible live service, we would instead randomize policies across fresh episodes.

ScienceWorld and greedy decoding. ScienceWorld counted completed tasks. Continuing succeeded on 19.30% of 57 tasks, and the direct forest intervened on only six of 51 eligible prefixes. On those six, its binary outcomes matched continuation on both draws. That explains why the policies tied under this endpoint. It does not diagnose why the actor struggled. We did not run temperature zero. With deterministic generation and environment transitions, both draws would coincide and the gap would be zero. On 91 additional ALFWorld tasks, raising temperature from 0.7 to 1.0 actually reduced the observed gap from 21.98 to 16.48 points. We report that comparison as exploratory in Appendix G.

Reviewer_UcjaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that runtime interventions for LLM agents (warn, replan, rollback) are often scored on the same rollout used to pick them, which inflates their value. The authors roll out every action twice from a saved state, pick on one rollout and score on the other, and freeze controller predictions beforehand. With Qwen3-8B on ALFWorld, same-rollout scoring overstates value by about 15 points, roughly what pure noise would produce. Random-forest controllers trained on these outcomes don’t significantly beat simply continuing, in either the scheduled-checkpoint study or the failure-triggered one.

Critical strengths

None

Critical weaknesses
  • The paper is nearly unreadable. The abstract cannot be understood without reading the paper, and the paper doesn’t help.
  • The paper repeatedly uses vague, AI-style wording (“room,” “reading,” “instrument,” “menu”) without context. These words carry the argument but never say anything concrete.
  • Sentences are packed with numbers from different denominators, and the two studies are tangled across sections and appendices.
  • The main finding is expected. Scoring the best of several noisy outcomes on the same outcomes is the winner’s curse, and the paper’s own permutation reference shows the gap is mostly noise.
  • The controller is a random forest on summary counts trained on noisy outcome labels. It has no task understanding and doesn’t use the agent’s reasoning or intermediate progress. No LLM-based controller is tried.
  • None of the controllers significantly beats continuing, but the abstract reads as positive.
  • Not acceptable in its current form. It needs a full rewrite so that a first-time reader can follow it.
Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ucja

We’re sorry we made you work so hard to understand the paper. In particular, we used “room” and “reading” where we should have said which action was chosen, which rollout was scored, and how many tasks entered the comparison. We rewrote the abstract and main text around one example. An agent tries to open a cabinet, the observation does not change, and the evaluator tries several repairs. If replanning succeeds once, what would justify calling it a rescue?

You are right that the winner’s curse is familiar. Our contribution is to measure it in this setting. On 384 ALFWorld tasks, the selected actions scored 69.40% on the rollouts that selected them and 54.04% when the same choices were scored on fresh rollouts. Shuffling the action and draw labels within each task gives an expected gap of 15.95 points, close to the observed 15.36. We now credit Double Q-learning in the introduction and give the absolute scores side by side in Table 2.

Your point about noise prompted a further check. In the six-action study, we asked whether an action that did relatively well on one draw tended to do relatively well again on the same task. It did, at both scheduled and failure-triggered checkpoints, even after accounting for task difficulty and average differences between actions. That leaves a meaningful distinction. The size of a repair’s advantage varied reproducibly, but our controller did not establish that it could use the interaction history to predict that advantage and improve on continuing. Section 5.3 gives the analysis, and the abstract now states the controller result explicitly. We also describe exactly which progress indicators and transcript features the forest received. An LLM controller remains an open comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.