- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes COTA, a runtime-intervention scheme for frozen LLM agents. Before the actor's proposed action is executed, K = 4 alternative actions are drawn from a reference candidate mechanism (legal environment actions, offline retrieval, or a 0.5B generator). A fine-tuned Qwen2.5-0.5B comparator judges each alternative against the proposal in both input orders. If at least R alternatives win, the top-ranked winner is shown to the actor as non-binding advice, and the actor replans. The comparator is trained on pairwise labels from same-prefix counterfactual branches: the environment is restored to a decision state, a different first action is executed, and the same frozen Qwen3-8B actor continues. Proposition 3.1 bounds the gap between the learned gate's domination estimate and the true domination rate by the comparator error plus a Hoeffding term. On WebShop, ALFWorld and τ³-Retail with three actors, COTA improves the original actor in all nine settings (e.g., WebShop 39.6 → 56.3 for Qwen3-8B), at 7.8–24× lower cost than AgentRM beam search.
Critical strengths
S1: COTA is cheap and practical. A 0.5B comparator, trained once on Qwen3-8B branches, substantially improves the source actor on WebShop (39.6 → 56.3) and ALFWorld (82.8 → 90.3), is reused without retraining for stronger actors, and costs 7.8–24× less than AgentRM beam search at 1.28× average runtime (Table 1). Prompts, costs and offline GPU-hours are fully documented (Appendices B.8, D).
S2: The diagnostics are informative and candidly reported. With the same 0.5B backbone and data, pairwise supervision clearly beats absolute-Q scoring (ALFWorld 57.5 → 90.3, Table 2). The cross-actor study (Table 13) openly shows that source-actor preferences mostly become ties under other actors, and the candidate-support study (Table 14) isolates the role of the candidate mechanism.
Critical weaknesses
W1: Table 2 contradicts the claim that the advice is constructive. Executing the comparator's winner (Pairwise/Forced) lowers Qwen3-8B from 82.8 to 51.5 on ALFWorld and from 37.5 to 16.7 on τ³-Retail. Applying the performance-difference lemma, I find that these winners have, on average, lower actor Q-value than the proposals they replace. Gains come from replanning; no random-advice control isolates the comparator.
W2: Several gains are within run-to-run noise. Table 3's B = 0 permits no intervention, yet scores 44.2 on τ³-Retail versus 37.5 for Original (Table 1); in episodes, I find full COTA adds one over it (54 vs 53 of 120). Results are single runs. For the 2–4-episode gains (Qwen3.6/DeepSeek τ³-Retail, DeepSeek ALFWorld), I computed best-case paired McNemar p ≥ 0.125.
W3: Proposition 3.1 is vacuous at the deployed K = 4. I computed its Hoeffding term at δ = 0.05 as 0.68, so the bound never certifies non-intervention for R = 1 and covers no state for R = 2. I also computed that a perfect comparator's R = 1 gate still fires 34% of the time at ρ = 0.1. The comparator error ε is never measured but maybe it's imoprtant somehow.
Questions for authors
- Forced execution runs exactly the branch procedure that defines the comparator's target (Section 3.3), so applying the performance-difference lemma, I find that the Forced drops in Table 2 imply the executed winners have lower actor Q-value than the proposals they replace. Do you agree, and if so, in what sense is the advice constructive?
- Can you add a control that keeps the replanning step but replaces the comparator with a random gate at the same per-step firing rate, advising a random candidate from the same mechanism? If this matches COTA, what does the comparator contribute?
- Why does B = 0 (no intervention) differ from Original on τ³-Retail (44.2 vs 37.5) and WebShop (36.6 vs 39.6)? Can you report several seeds with confidence intervals and paired per-episode comparisons?
- How does COTA differ from small step-level value models that rank candidate actions for larger frozen agents on WebShop and ALFWorld, such as a 1.3B Q-value model trained on same-state action pairs (Zhai et al., https://arxiv.org/abs/2409.09345) or the small offline critic of Retrospex (Xiang et al., https://arxiv.org/abs/2505.11807)? Can one of them, scoring the same K candidates at matched cost, be added as a baseline?
8 comments
Rebuttal by Authors [2/3]
Q2: Random gate with random advice at the same firing rate.
Thank you for this suggestion. It became the most informative experiment in the revision. We ran your control on the same 500 WebShop tasks with matched seeds (Qwen3), together with intermediate conditions that separate when COTA intervenes from what it advises (, on p. 22, summarized in ). One clarification frames the results: COTA's gate is not a separate module but the learned gate of Eq. 10, i.e., the rule applied to the 0.5B comparator's judgments (, "COTA gate" in ). The comparator judges each of the sampled alternatives against the proposal, and fires when at least of them are predicted to win, i.e., when the estimated domination rate reaches . The highest-ranked winner then becomes the advice, and the actor decides through replanning whether to follow it.
Full COTA scores 54.7, against 34.7 for your control (random gate + random advice): a gain of +20.0. Your control itself falls below Original (37.2), so an extra replanning opportunity is not useful by itself. The gain splits into two parts:
- When to intervene (+15.3). Keeping the advice random and replacing only the random gate by raises the reward from 34.7 to 50.0. The two conditions fire at almost the same first-review rate (36.9% vs. 36.2%), so frequency cannot explain this gain: it comes from the comparator choosing the right states.
- What to advise (+4.73). At an essentially identical firing rate (36.8% vs. 36.9%), comparator-selected advice raises the reward further, from 50.0 to 54.7.
This also bears on W3. Individually, the comparator's selected winners agree with rollout preferences in only 67.6–72.2% of held-out cases (), yet within COTA's gate and replanning loop it becomes highly effective. Gating with and no advice at all provides supporting evidence: its firing rate (42.6%) is not matched, but even a bare request to reconsider at the comparator's chosen states scores 40.5, above Original (37.2).
Taken together with W1, no single component suffices: replanning at random states with random advice falls below Original, and executing the comparator's choice directly falls below Original on ALFWorld and Retail (). COTA's gains come from their combination: the rule turns the comparator's pairwise judgments into the decision of when to intervene, its top-ranked winner supplies the advice, and non-binding replanning lets the actor turn that advice into its own action.
W2 & Q3: Why did differ from Original? Are the gains within noise?
Thank you for checking the episode counts. The discrepancy came only from rollout seeds: the WebShop/Retail runs used a different seed from , while Full reused the main-table COTA run. We have rerun all Qwen settings in with five seeds, except the single-run AgentRM baselines, and Original/ and COTA/Full are now identical across (pp. 7 and 9). COTA improves Original by 15.9/6.72/5.17 points with Qwen3 and 10.4/6.42/4.50 points with Qwen3.6 on WebShop/ALFWorld/Retail.
( on p. 23) now reports two 95% confidence intervals for every gain, together with paired per-episode McNemar tests. Seed-level intervals are computed over the five per-seed differences and capture run-to-run variability. Task-paired intervals treat the task as the unit of analysis, so repeated evaluations of the same task are not counted as independent. On WebShop and ALFWorld, both intervals exclude zero and the McNemar tests are significant for both actors (e.g., Qwen3 WebShop: seed-level [14.4, 17.5], task-paired [13.3, 18.5]). Retail, with only 40 tasks, is the lowest-powered benchmark. Its gains are positive for both actors, but some of its intervals include zero (lower bounds no lower than −0.290). You are right about the small single-run DeepSeek gains: on ALFWorld (36/40 → 40/40) and Retail (97/120 → 99/120), the best-case paired McNemar is 0.125 and 0.500, so we report them as descriptive. The replicable evidence is the five-seed gains on WebShop and ALFWorld for both Qwen actors.
Rebuttal by Authors [3/3]
W3: Proposition 3.1 at , the 34% firing rate, and the comparator error .
We agree with your arithmetic: the Hoeffding term is 0.679 at , . Three clarifications follow.
- Proposition 3.1 is an error decomposition, not a per-decision certificate. It shows that the gate's error consists of comparator error and finite-candidate error, and it holds for every . This decomposition motivates the design: same-prefix supervision with an explicit tie class reduces , while set the threshold and trade candidate cost against sampling error. As with classical generalization bounds, which are vacuous for practical networks yet explain what drives error [2], the worst-case constant is conservative. We therefore select on held-out end-to-end performance (), and now describes as a threshold on the empirical estimate rather than as a statistical test.
- For a perfect comparator, the 34% firing rate is a discovery rate, not an error rate. With independent sampling, , , and , a perfect comparator fires with probability , while the population-level gate stays silent (). This divergence is precisely what the finite-candidate term of Proposition 3.1 accounts for. At , the oracle gate fires exactly when a sampled alternative is truly better than the proposal, so every such intervention returns genuinely superior advice. This is the intended behavior.
- We now report how reliable the comparator's selected winners are (, on p. 20). On held-out pairs, 70.0%, 72.2%, and 67.6% of the winners selected by the comparator agree with the rollout preference on WebShop, ALFWorld, and Retail. Only 17.1–18.4% are outright reversals. The rest are rollout ties. This selected-winner precision does not count missed winners, which the full of Proposition 3.1 also includes. COTA is designed for exactly such an imperfect comparator, and its gate is most reliable where it matters. A clearly bad proposal has a high domination rate, so most sampled alternatives beat it, and with the gate needs to detect only one of these wins among the candidates. Even with substantial per-comparison error, it rarely misses all of them. A false fire, in turn, is cheap: the advice is non-binding, and DeepSeek keeps its proposal in 49.6–63.6% of replan events (). Q2 confirms this empirically: at a matched firing rate, replacing the random gate with is worth +15.3 points.
Q4: Comparison with small step-level value models [3, 4].
Thank you for pointing us to these closely related works. We now discuss both in and add both as baselines in (p. 7), with implementation details in . Each scores the same candidates inside COTA's gate-and-advice interface, so only the learned scorer differs. Retrospex-IQL [4] falls below Original in all four evaluated settings, and COTA exceeds it by 12.5–26.6 points at lower cost (USD 0.383–1.08 vs. 0.651–1.87 per 100 episodes). The critic of Zhai et al. [3], run on WebShop (the only benchmark with released training data), improves Original by just 3.70 points for Qwen3 and is on par for Qwen3.6 (41.0 and 56.6 vs. COTA's 53.2 and 66.9), at over twice COTA's cost and 12.3–12.8× Original runtime (COTA: 1.11–1.68×). The difference lies in how actions are scored: both baselines give each action its own scalar value: Retrospex-IQL regresses absolute -values by TD learning, and [3] scores actions through an implicit -value learned by step-level DPO. COTA's comparator instead judges the proposal and an alternative jointly under the shared prefix, so the 0.5B model only needs the sign of a local difference. Our own absolute- ablation on the same backbone and data () isolates the objective: with constructive advice, pairwise comparison beats absolute- in all six settings (e.g., 90.3 vs. 82.1 on Qwen3 ALFWorld).
[2] Dziugaite & Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. UAI 2017.
[3] Zhai et al. Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models. AAAI 2025.
[4] Xiang et al. Retrospex: Language Agent Meets Offline Reinforcement Learning Critic. EMNLP 2024.
We hope these controls resolve your concerns. Your Q2 control in particular turned an open question into the clearest evidence for COTA's design. If they do, we would be grateful if you would consider raising your score, and we are glad to answer any further questions.
Rebuttal by Authors [1/3]
Thank you for the careful, technical review. Your performance-difference argument and the random-advice control you proposed go to the central question about COTA: what does the comparator contribute beyond giving the actor a second chance? We ran your control, and it answers this question directly. Below we respond to the Weaknesses (W) and Questions (Q).
W1 & Q1: Do the Forced drops imply that the executed winners have lower actor Q-value?
We agree with your reasoning, and it is precisely why COTA never executes the comparator's choice directly. Two observations complete the picture.
[1] Torrey & Taylor. Teaching on a Budget: Agents Advising Agents in Reinforcement Learning. AAMAS 2013.