- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper is about doing runtime intervention for LLM agents, ie we put an LLM agent (Qwen3-8B) into a multi-step env (ALFWorld, with ScienceWorld as a small boundary panel), stop at one point in the trajectory and predict what to insert: continue, warn, replan, rollback (and in the second study also a three-step rollback and a “name what you still need and where you haven’t looked” message). From the same saved state, they run every action twice with independent seeds, so an action can be chosen on one draw and scored on the other, and they show that the usual choose-and-score-on-the-same-branch evaluation overstates value by 15.36 points on 384 tasks, which doesn’t exceed what you get by shuffling action labels within each task (17.26 on eligible prefixes), so it’s basically continuation noise.
On top of these branches the author also fit random forests on some hand engineered features (counts of invalid actions, reward summaries, TF-IDF of recent text etc) to predict either each arm’s outcome or its benefit over continuing, plus a “cross-draw” version that picks the arm with one draw’s forest and values it with the other’s, and all predictions are frozen before any arm outcome exists. In the first study (decision at a fixed early step) no controller beats continuation significantly and none beats always-replan, and in the second (decision at the first invalid action or unchanged observation, 593 tasks) the cross-draw controller gets 56.24% vs 54.97% for continue and 54.30% for the best fixed repair, but neither gap passes their pre-registered test.
Critical strengths
None
Critical weaknesses
-
The setting itself is sound, however the execution and experiment setup is less thought out. For example, why do we need a “warn” in a selection choice when no obvious context or signal requires it? Also the study is conducted with only single LLM in a single experiment environment + limited settings environment. I personally highly doubt the observations found so far to be even generalizable to broad LLM behavior or whether it's just because the model is too small, suppose we scale the model size does behave better? What about LLMs equipped with agentic behavior such as Ornith 9B?
-
Based on the experiment results, I’m not convinced that the learned controller falls to select “always repair”, which is obvious in hindsight. If I have to blindly guess from 4 interrupt keyphrases : continue, warn, replan, rollback, of course I would choose replan, same like S1 [arxiv:2501.19393] where you just add the same keyphrase (“Wait”) to get a long CoT.
Questions for authors
-
Why is a random forest the initial choice and not a linear model? It seems odd that this is the case, as far as I can see there’s no linear or logistic model anywhere in the paper and citing Breiman is the only justification. The only alternatives are on p.16 (gradient boosting, 0.6B LM embeddings, and both together), at 29.27/29.10/28.84 vs 29.47 at the scheduled point and 49.46/50.29/50.15 vs 50.69 at the triggered one. These are all within about a point, so I’m not sure I can read that as “model class doesn’t matter”, it may just be within dev noise (also, why is scheduled dev utility 29.47% when confirmation success is around 55%?).
-
The main text says 593 tasks run under both rules (497 and 540 eligible), but Appendix D and Table 11 say both rules run the same 288 identities, and 593 can’t be balanced over six task types either. Which is right?
-
One missing discussion which is what is the connect of this setup to contextual bandits? Since contextual bandits is basically : “pick one of K actions given context” but the paper never compares to or discusses them. I get that with every arm run twice on every dev prefix the data is full information rather than bandit feedback, but that makes a simple per-arm model even easier to fit, it pools across tasks the same way your Efron & Morris argument wants, and it’s the natural thing if this were deployed on a stream of incoming tasks where you only see the arm you fired.
3 comments
We also ran the per-action regression you suggested as LR
However, given that logistic regression and random forests achieve almost identical performance (53.91% vs. 54.04%), I think a more fundamental question remains: what is the learned controller actually learning, and why should we expect it to work?
Does this similarity suggest that the intervention decision depends on relatively simple features, or that the available features contain too little signal to reliably distinguish between interventions? In particular, it would be helpful to understand whether the controller learns meaningful context-dependent intervention choices or primarily reproduces a preference for certain actions.
Since neither learned controller demonstrates a statistically significant improvement over the fixed baselines, I still find it difficult to distinguish genuine intervention-selection ability from noise. The additional regression baselines clarify that model complexity is probably not the main issue, but they do not yet explain whether the proposed learning approach captures useful intervention signals.
The learned rule did not collapse to replanning. It continued on 245 of 355 eligible prefixes, replanned on 106, and warned on four.
While the action distribution is useful, it does not necessarily demonstrate that the controller has learned when to intervene. Could the authors provide evidence that the selected interventions outperform alternative actions on the particular states where they are chosen? Since all intervention outcomes are available in the experimental setup, a conditional analysis of intervention gains would help distinguish meaningful context-dependent selection from simply learning an overall action preference.
separate 27B experiment, but it also changed the policy, filtering, and decoding. It cannot tell us how the same intervention controller would behave if only actor size changed.
However, this leaves my original concern about generalizability largely unresolved. In particular, how much of the observed difficulty in selecting effective interventions is attributable to the controller, and how much is specific to the Qwen3-8B actor? Without a controlled comparison across actor capabilities, it seems difficult to establish whether the reported conclusions extend beyond this particular agent configuration.
Does the controller choose the right repair for the state?
Your question points to a comparison we should have reported. The logistic model and the outcome forest chose the same action on only 142 of 355 eligible ALFWorld states (40%), despite scoring 53.91% and 54.04% over all planned tasks. Similar success rates therefore tell us little about which rule either model learned. To test whether the choices were useful, we went back to the two held-out outcomes for every action. The controllers’ choices had been recorded before those outcomes were generated.
Where did the chosen repair help? In Study 1, on the 106 states where the direct forest requested a new plan, replanning succeeded 61.32% of the time and continuing succeeded 55.66%. But replanning also beat continuing by 1.43 points on the 245 states where the forest chose to continue. The difference between those gains was 4.23 points (95% interval −2.58 to 11.13). On this panel, we cannot distinguish reliable targeting from an overall preference for replanning.
Study 2 gives a more informative result. At a failure signal, the controller sometimes asked the agent to name the objects it still needed and the places it had not searched. On the 163 states where it chose that reminder, the reminder scored 45.09% versus 40.49% for continuing. On the 298 states where it chose to continue, the same comparison went the other way: 49.16% versus 50.84%. The reminder’s relative gain differed by 6.28 points between those groups (95% interval 0.30 to 12.32). The remaining 36 states received a warning.
We also tested your action-preference explanation directly. We kept the exact frequencies of all chosen actions and calculated what success would have been, on average, if those same choices had been assigned at random across eligible states. At failure signals, the frozen assignments scored 50.40% versus 48.64% for that baseline, a gain of 1.76 points (0.34 to 3.17). Holding the mix fixed within each of the six task types still gave +1.61 points (0.21 to 2.94). Holding it fixed within each floorplan gave +0.70 points (−0.66 to 2.04), so the data do not resolve how much the controller picks up within a layout. At scheduled checkpoints the same frozen controller gained −0.07 points over the randomly assigned mix (−1.62 to 1.47). These are retrospective comparisons with pointwise 95% intervals from 50,000 whole-floorplan bootstrap resamples. They use eligible states only. The planned comparison against always continuing still includes zero.
Our reading is that the failure-triggered controller picked up some information about where the reminder was useful beyond how often it preferred that action. The data do not identify which prefix feature carried that information. On your actor-size question, we agree that the 27B experiment cannot separate actor capability from controller quality. We would need the same checkpoints, repairs and budget with both actors, testing the frozen controller on each and then refitting it for each actor. What transfers without that experiment is the evaluation procedure. On these data it distinguishes three empirically different things: a rollout that makes a repair look good in hindsight, a repair that tends to help on the states where the controller chooses it, and a controller that beats a fixed policy over whole episodes. We found evidence for the second at failure signals. The third remains open.
Response to Reviewer Sz7K
We’re sorry for the contradiction you found in the task counts. 593 is the correct number under each checkpoint rule. The first 288 identities were balanced, with 48 in each of six task types. We then added 305 identities from the remaining inventory. That combined set was unbalanced. Of the 593 tasks, 497 reached an event checkpoint and 540 reached a scheduled checkpoint, with 487 eligible under both rules. We checked the identities against the recorded runs and corrected the main text, Appendix D, and the table captions.
Your fixed-prompt comparison was well chosen. The warning asks the actor to check its goal, latest observation, and previous action. At a scheduled step it can interrupt an agent that is doing fine. Always replanning was therefore an important baseline, and on Study 1 ALFWorld it scored 55.34%, ahead of the learned direct forest at 54.43% and continuing at 52.86%. The learned rule did not collapse to replanning. It continued on 245 of 355 eligible prefixes, replanned on 106, and warned on four. Study 2 gave a different answer. At failure signals the controller continued on 298 prefixes, warned on 36, and asked the actor to locate missing objects on 163. It never chose replanning there. Section 5 now reports what the controllers actually did.
We also ran the per-action regression you suggested. Your contextual-bandit analogy describes the execution decision well. During training, restoring each development prefix gave us two observed outcomes for every available action, so we could fit one logistic model per action directly. On held-out Study 1 ALFWorld tasks, logistic regression scored 53.91% versus 54.04% for the outcome forest. The paired difference was −0.13 points, with a 95% interval from −2.66 to 2.41. Ridge regression on gains over continuing scored 54.30% versus 54.43% for the direct forest. We agree that the small differences among boosting, embeddings, and forests on development data could not settle the question of model capacity. Appendix J now reports the held-out linear comparisons and their uncertainty.
On actor size, the planned studies used Qwen3-8B. We do have a separate 27B experiment, but it also changed the policy, filtering, and decoding. It cannot tell us how the same intervention controller would behave if only actor size changed. We have made that distinction explicit in Appendix G.