acceptodds
Under review as a conference paper at ICLR 2027

Don’t Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents

Abstract

LLM agents are emerging as an important paradigm for real-world tasks that require reasoning, tool use, and sequential decision-making. As these agents operate over longer horizons, runtime intervention offers a way to improve reliability without retraining the underlying actor. Effective intervention must provide a useful direction for recovery besides a warning. Existing approaches often rely on an expert solver or a critic that generates task-specific corrections, incurring either the cost of another capable solver or the capacity demands of a task-capable critic. We introduce omparison-nly iny dvisor () for constructive runtime intervention, which reduces the learned intervention role to local action comparison. A lightweight comparator judges the actor's proposal against available alternatives, and preferred alternatives are returned as non-binding advice for replanning. The comparator is trained from same-prefix counterfactual branches. Across WebShop, ALFWorld, and -Retail with three LLM actors, COTA instantiated with a 0.5B comparator consistently improves the original actor and achieves the strongest overall performance–cost trade-off among the compared methods. These results suggest that effective runtime intervention need not itself be a task-solving problem: the intervention role can be separated from task solving and handled by a lightweight model specialized for local comparison.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Model Agents, Runtime Intervention, Agent Reliability, Weak-for-Strong Runtime Assistance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4z7NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes COTA, a runtime-intervention scheme for frozen LLM agents. Before the actor's proposed action is executed, K = 4 alternative actions are drawn from a reference candidate mechanism (legal environment actions, offline retrieval, or a 0.5B generator). A fine-tuned Qwen2.5-0.5B comparator judges each alternative against the proposal in both input orders. If at least R alternatives win, the top-ranked winner is shown to the actor as non-binding advice, and the actor replans. The comparator is trained on pairwise labels from same-prefix counterfactual branches: the environment is restored to a decision state, a different first action is executed, and the same frozen Qwen3-8B actor continues. Proposition 3.1 bounds the gap between the learned gate's domination estimate and the true domination rate by the comparator error plus a Hoeffding term. On WebShop, ALFWorld and τ³-Retail with three actors, COTA improves the original actor in all nine settings (e.g., WebShop 39.6 → 56.3 for Qwen3-8B), at 7.8–24× lower cost than AgentRM beam search.

Critical strengths

S1: COTA is cheap and practical. A 0.5B comparator, trained once on Qwen3-8B branches, substantially improves the source actor on WebShop (39.6 → 56.3) and ALFWorld (82.8 → 90.3), is reused without retraining for stronger actors, and costs 7.8–24× less than AgentRM beam search at 1.28× average runtime (Table 1). Prompts, costs and offline GPU-hours are fully documented (Appendices B.8, D).

S2: The diagnostics are informative and candidly reported. With the same 0.5B backbone and data, pairwise supervision clearly beats absolute-Q scoring (ALFWorld 57.5 → 90.3, Table 2). The cross-actor study (Table 13) openly shows that source-actor preferences mostly become ties under other actors, and the candidate-support study (Table 14) isolates the role of the candidate mechanism.

Critical weaknesses

W1: Table 2 contradicts the claim that the advice is constructive. Executing the comparator's winner (Pairwise/Forced) lowers Qwen3-8B from 82.8 to 51.5 on ALFWorld and from 37.5 to 16.7 on τ³-Retail. Applying the performance-difference lemma, I find that these winners have, on average, lower actor Q-value than the proposals they replace. Gains come from replanning; no random-advice control isolates the comparator.

W2: Several gains are within run-to-run noise. Table 3's B = 0 permits no intervention, yet scores 44.2 on τ³-Retail versus 37.5 for Original (Table 1); in episodes, I find full COTA adds one over it (54 vs 53 of 120). Results are single runs. For the 2–4-episode gains (Qwen3.6/DeepSeek τ³-Retail, DeepSeek ALFWorld), I computed best-case paired McNemar p ≥ 0.125.

W3: Proposition 3.1 is vacuous at the deployed K = 4. I computed its Hoeffding term at δ = 0.05 as 0.68, so the bound never certifies non-intervention for R = 1 and covers no state for R = 2. I also computed that a perfect comparator's R = 1 gate still fires 34% of the time at ρ = 0.1. The comparator error ε is never measured but maybe it's imoprtant somehow.

Questions for authors
  1. Forced execution runs exactly the branch procedure that defines the comparator's target (Section 3.3), so applying the performance-difference lemma, I find that the Forced drops in Table 2 imply the executed winners have lower actor Q-value than the proposals they replace. Do you agree, and if so, in what sense is the advice constructive?
  2. Can you add a control that keeps the replanning step but replaces the comparator with a random gate at the same per-step firing rate, advising a random candidate from the same mechanism? If this matches COTA, what does the comparator contribute?
  3. Why does B = 0 (no intervention) differ from Original on τ³-Retail (44.2 vs 37.5) and WebShop (36.6 vs 39.6)? Can you report several seeds with confidence intervals and paired per-episode comparisons?
  4. How does COTA differ from small step-level value models that rank candidate actions for larger frozen agents on WebShop and ALFWorld, such as a 1.3B Q-value model trained on same-state action pairs (Zhai et al., https://arxiv.org/abs/2409.09345) or the small offline critic of Retrospex (Xiang et al., https://arxiv.org/abs/2505.11807)? Can one of them, scoring the same K candidates at matched cost, be added as a baseline?
8 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/3]

Thank you for the careful, technical review. Your performance-difference argument and the random-advice control you proposed go to the central question about COTA: what does the comparator contribute beyond giving the actor a second chance? We ran your control, and it answers this question directly. Below we respond to the Weaknesses (W) and Questions (Q).

W1 & Q1: Do the Forced drops imply that the executed winners have lower actor QQ-value?

We agree with your reasoning, and it is precisely why COTA never executes the comparator's choice directly. Two observations complete the picture.

  1. The submitted Forced condition conflated the interface with intervention frequency. It substituted the top-ranked alternative at nearly every reviewed step, including states where the comparator itself predicted no winner, so the performance-difference sum averaged over many states where the proposal was already good. In the revised ablation (Section 4.4\color{blue}{\textbf{Section 4.4}}, Table 2\color{blue}{\textbf{Table 2}} on p. 8), Forced uses a random gate calibrated to COTA's step-level rate. We apply the same frequency control to absolute-QQ constructive intervention, whose absolute scores would otherwise fire far more often than COTA on the binary-outcome benchmarks (ALFWorld and Retail). Under this control, Qwen3 recovers from 51.5 to 76.1 on ALFWorld and from 16.7 to 36.7 on Retail. On WebShop, executing the comparator's choice at random states already improves over Original on the same seed (39.6 and 56.6) by 7.10 (Qwen3) and 6.90 (Qwen3.6) points, so there the average advantage of the executed winners is positive.
  2. Where forced execution still trails Original (ALFWorld, Retail), your conclusion holds, and COTA is built around it. A 0.5B comparator's top choice is not uniformly better when executed blindly. COTA therefore (i) fires only when at least RR of the KK sampled alternatives are predicted to beat the proposal, i.e., when the estimated domination rate reaches R/KR/K, and (ii) returns the winner as advice, so the executed action is the actor's own replanned decision. Your argument applies to the forced winners, not to COTA's executed actions. Used constructively, the same comparator raises Qwen3 from 76.1 to 90.3 on ALFWorld and from 36.7 to 45.0 on Retail. The actor acts as a second filter: as Figure 1\color{blue}{\textbf{Figure 1}} shows, DeepSeek adopts only 29.2–32.9% of the advice and keeps its proposal in 49.6–63.6% of replan events. The advice is "constructive" in the sense established for action advising in RL [1]: an imperfect teacher improves the learner by pointing to a better region of the action space, while the learner retains the decision.

[1] Torrey & Taylor. Teaching on a Budget: Agents Advising Agents in Reinforcement Learning. AAMAS 2013.

AuthorsOpenReview ↗

Rebuttal by Authors [2/3]

Q2: Random gate with random advice at the same firing rate.

Thank you for this suggestion. It became the most informative experiment in the revision. We ran your control on the same 500 WebShop tasks with matched seeds (Qwen3), together with intermediate conditions that separate when COTA intervenes from what it advises (Appendix C.5\color{blue}{\textbf{Appendix C.5}}, Table 15\color{blue}{\textbf{Table 15}} on p. 22, summarized in Section 4.4\color{blue}{\textbf{Section 4.4}}). One clarification frames the results: COTA's gate is not a separate module but the learned gate gθg_\theta of Eq. 10, i.e., the R/KR/K rule applied to the 0.5B comparator's judgments (Sections 3.2 and 3.4\color{blue}{\textbf{Sections 3.2 and 3.4}}, "COTA gate" in Table 15\color{blue}{\textbf{Table 15}}). The comparator judges each of the KK sampled alternatives against the proposal, and gθg_\theta fires when at least RR of them are predicted to win, i.e., when the estimated domination rate reaches R/KR/K. The highest-ranked winner then becomes the advice, and the actor decides through replanning whether to follow it.

Full COTA scores 54.7, against 34.7 for your control (random gate + random advice): a gain of +20.0. Your control itself falls below Original (37.2), so an extra replanning opportunity is not useful by itself. The gain splits into two parts:

  • When to intervene (+15.3). Keeping the advice random and replacing only the random gate by gθg_\theta raises the reward from 34.7 to 50.0. The two conditions fire at almost the same first-review rate (36.9% vs. 36.2%), so frequency cannot explain this gain: it comes from the comparator choosing the right states.
  • What to advise (+4.73). At an essentially identical firing rate (36.8% vs. 36.9%), comparator-selected advice raises the reward further, from 50.0 to 54.7.

This also bears on W3. Individually, the comparator's selected winners agree with rollout preferences in only 67.6–72.2% of held-out cases (Table 10\color{blue}{\textbf{Table 10}}), yet within COTA's R/KR/K gate and replanning loop it becomes highly effective. Gating with gθg_\theta and no advice at all provides supporting evidence: its firing rate (42.6%) is not matched, but even a bare request to reconsider at the comparator's chosen states scores 40.5, above Original (37.2).

Taken together with W1, no single component suffices: replanning at random states with random advice falls below Original, and executing the comparator's choice directly falls below Original on ALFWorld and Retail (Table 2\color{blue}{\textbf{Table 2}}). COTA's gains come from their combination: the R/KR/K rule turns the comparator's pairwise judgments into the decision of when to intervene, its top-ranked winner supplies the advice, and non-binding replanning lets the actor turn that advice into its own action.

W2 & Q3: Why did B=0B=0 differ from Original? Are the gains within noise?

Thank you for checking the episode counts. The discrepancy came only from rollout seeds: the WebShop/Retail B=0,1,2B=0,1,2 runs used a different seed from Table 1\color{blue}{\textbf{Table 1}}, while Full reused the main-table COTA run. We have rerun all Qwen settings in Tables 1 and 3\color{blue}{\textbf{Tables 1 and 3}} with five seeds, except the single-run AgentRM baselines, and Original/B=0B=0 and COTA/Full are now identical across Tables 1 and 3\color{blue}{\textbf{Tables 1 and 3}} (pp. 7 and 9). COTA improves Original by 15.9/6.72/5.17 points with Qwen3 and 10.4/6.42/4.50 points with Qwen3.6 on WebShop/ALFWorld/Retail.

Appendix C.6\color{blue}{\textbf{Appendix C.6}} (Table 16\color{blue}{\textbf{Table 16}} on p. 23) now reports two 95% confidence intervals for every gain, together with paired per-episode McNemar tests. Seed-level intervals are computed over the five per-seed differences and capture run-to-run variability. Task-paired intervals treat the task as the unit of analysis, so repeated evaluations of the same task are not counted as independent. On WebShop and ALFWorld, both intervals exclude zero and the McNemar tests are significant for both actors (e.g., Qwen3 WebShop: seed-level [14.4, 17.5], task-paired [13.3, 18.5]). Retail, with only 40 tasks, is the lowest-powered benchmark. Its gains are positive for both actors, but some of its intervals include zero (lower bounds no lower than −0.290). You are right about the small single-run DeepSeek gains: on ALFWorld (36/40 → 40/40) and Retail (97/120 → 99/120), the best-case paired McNemar pp is 0.125 and 0.500, so we report them as descriptive. The replicable evidence is the five-seed gains on WebShop and ALFWorld for both Qwen actors.

AuthorsOpenReview ↗

Rebuttal by Authors [3/3]

W3: Proposition 3.1 at K=4K=4, the 34% firing rate, and the comparator error ϵ\epsilon.

We agree with your arithmetic: the Hoeffding term is 0.679 at K=4K=4, δ=0.05\delta=0.05. Three clarifications follow.

  1. Proposition 3.1 is an error decomposition, not a per-decision certificate. It shows that the gate's error consists of comparator error and finite-candidate error, and it holds for every KK. This decomposition motivates the design: same-prefix supervision with an explicit tie class reduces ϵ\epsilon, while (K,R)(K, R) set the threshold R/KR/K and trade candidate cost against sampling error. As with classical generalization bounds, which are vacuous for practical networks yet explain what drives error [2], the worst-case constant is conservative. We therefore select (K,R)(K, R) on held-out end-to-end performance (Appendix B.5\color{blue}{\textbf{Appendix B.5}}), and Section 3.4\color{blue}{\textbf{Section 3.4}} now describes R/KR/K as a threshold on the empirical estimate rather than as a statistical test.
  2. For a perfect comparator, the 34% firing rate is a discovery rate, not an error rate. With independent sampling, ρ=0.1\rho=0.1, K=4K=4, and R=1R=1, a perfect comparator fires with probability 1−0.94=34.4%1-0.9^4=34.4\%, while the population-level gate I[ρ≥R/K]\mathbb{I}[\rho\geq R/K] stays silent (0.1<0.250.1<0.25). This divergence is precisely what the finite-candidate term of Proposition 3.1 accounts for. At ρ=0.1\rho=0.1, the oracle R=1R=1 gate fires exactly when a sampled alternative is truly better than the proposal, so every such intervention returns genuinely superior advice. This is the intended behavior.
  3. We now report how reliable the comparator's selected winners are (Appendix C.1\color{blue}{\textbf{Appendix C.1}}, Table 10\color{blue}{\textbf{Table 10}} on p. 20). On held-out pairs, 70.0%, 72.2%, and 67.6% of the winners selected by the comparator agree with the rollout preference on WebShop, ALFWorld, and Retail. Only 17.1–18.4% are outright reversals. The rest are rollout ties. This selected-winner precision does not count missed winners, which the full ϵ\epsilon of Proposition 3.1 also includes. COTA is designed for exactly such an imperfect comparator, and its gate is most reliable where it matters. A clearly bad proposal has a high domination rate, so most sampled alternatives beat it, and with R=1R=1 the gate needs to detect only one of these wins among the K=4K=4 candidates. Even with substantial per-comparison error, it rarely misses all of them. A false fire, in turn, is cheap: the advice is non-binding, and DeepSeek keeps its proposal in 49.6–63.6% of replan events (Figure 1\color{blue}{\textbf{Figure 1}}). Q2 confirms this empirically: at a matched firing rate, replacing the random gate with gθg_\theta is worth +15.3 points.

Q4: Comparison with small step-level value models [3, 4].

Thank you for pointing us to these closely related works. We now discuss both in Section 5\color{blue}{\textbf{Section 5}} and add both as baselines in Table 1\color{blue}{\textbf{Table 1}} (p. 7), with implementation details in Appendix B.7\color{blue}{\textbf{Appendix B.7}}. Each scores the same KK candidates inside COTA's gate-and-advice interface, so only the learned scorer differs. Retrospex-IQL [4] falls below Original in all four evaluated settings, and COTA exceeds it by 12.5–26.6 points at lower cost (USD 0.383–1.08 vs. 0.651–1.87 per 100 episodes). The critic of Zhai et al. [3], run on WebShop (the only benchmark with released training data), improves Original by just 3.70 points for Qwen3 and is on par for Qwen3.6 (41.0 and 56.6 vs. COTA's 53.2 and 66.9), at over twice COTA's cost and 12.3–12.8× Original runtime (COTA: 1.11–1.68×). The difference lies in how actions are scored: both baselines give each action its own scalar value: Retrospex-IQL regresses absolute QQ-values by TD learning, and [3] scores actions through an implicit QQ-value learned by step-level DPO. COTA's comparator instead judges the proposal and an alternative jointly under the shared prefix, so the 0.5B model only needs the sign of a local difference. Our own absolute-QQ ablation on the same backbone and data (Table 2\color{blue}{\textbf{Table 2}}) isolates the objective: with constructive advice, pairwise comparison beats absolute-QQ in all six settings (e.g., 90.3 vs. 82.1 on Qwen3 ALFWorld).

[2] Dziugaite & Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. UAI 2017.

[3] Zhai et al. Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models. AAAI 2025.

[4] Xiang et al. Retrospex: Language Agent Meets Offline Reinforcement Learning Critic. EMNLP 2024.

We hope these controls resolve your concerns. Your Q2 control in particular turned an open question into the clearest evidence for COTA's design. If they do, we would be grateful if you would consider raising your score, and we are glad to answer any further questions.

Reviewer_TYYPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COTA, a runtime advisor that uses a 0.5B model to compare an agent’s proposed action with available alternatives. When enough alternatives are preferred, the agent receives advice and replans rather than being forced to follow it. The comparator is trained on branches from the same trajectory prefix, and the paper provides a statistical analysis of the intervention gate. Experiments on WebShop, ALFWorld, and Retail with three actors report performance gains, supported by ablations of the learning objective, intervention mechanism, and candidate sources.

Critical strengths

The paper presents a practical way to use a small comparator to guide a larger agent without replacing its decision process. Table 2 supports the value of combining pairwise comparison with non-binding advice, especially on ALFWorld.

The performance gains on WebShop and ALFWorld are substantial, and the comparator is reused without retraining for stronger actors (Table 1 and Section 4.3). This suggests that the approach can provide useful assistance beyond the actor used to collect its training data.

Critical weaknesses
  1. The inconsistent baselines make some gains unclear. Retail’s original actor scores 37.5% in Table 1, but the no-intervention setting scores 44.2% in Table 3, against COTA’s 45.0%. This changes the apparent benefit substantially, and paired uncertainty estimates are needed to assess it.

  2. The cost comparison does not consistently account for auxiliary models. Appendix B.8 excludes the local 0.5B comparator’s monetary cost while assigning a proxy cost to AgentRM’s reward model. Since efficiency is a central contribution, the claimed performance–cost advantage needs validation under consistent accounting.

Questions for authors
  1. Clarification: Why do the original actor results in Table 1 differ from the no-intervention results in Table 3, especially on Retail? Please explain any changes in seeds or settings and provide paired confidence intervals that account for repeated tasks across seeds.

  2. Validation of the current claim: Does COTA retain its performance–cost advantage when the comparator and candidate-generation costs are included using a consistent accounting method across systems? This would help assess the strength of the efficiency contribution.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the focused review, and for recognizing COTA as a practical way for a small comparator to guide a larger agent without replacing its decision process. Both of your concerns ask whether the gains and the efficiency claim survive rigorous accounting. We reran the evaluation and recomputed costs to answer them directly. Below we respond to the Weaknesses (W) and Questions (Q).

W1 & Q1: Inconsistent baselines and paired uncertainty.

We appreciate you catching this. The discrepancy came only from rollout seeds: the WebShop/Retail B=0,1,2B=0,1,2 entries used a different seed from Table 1\color{blue}{\textbf{Table 1}}, while Full reused the main-table COTA run. Tasks, prompts, and decoding parameters were unchanged. We have replaced all Qwen results in Tables 1 and 3\color{blue}{\textbf{Tables 1 and 3}}, except the single-run AgentRM baselines, with five-seed runs, so Original/B=0B=0 and COTA/Full are now identical across Tables 1 and 3\color{blue}{\textbf{Tables 1 and 3}} (pp. 7 and 9). On Retail, Original is 39.8 and COTA is 45.0 for Qwen3, and 67.2 vs. 71.7 for Qwen3.6.

As requested, Appendix C.6\color{blue}{\textbf{Appendix C.6}} (Table 16\color{blue}{\textbf{Table 16}} on p. 23) now reports 95% confidence intervals paired by task: each task, not each episode, is the unit of analysis, so repeated evaluations of a task across seeds are not treated as independent. Alongside them, it gives seed-level intervals over the five per-seed differences, which capture run-to-run variability, and paired per-episode McNemar tests. On WebShop and ALFWorld, both intervals exclude zero and the McNemar tests are significant for both actors. The task-paired intervals are [13.3, 18.5] and [0.840, 12.6] for Qwen3, and [8.40, 12.4] and [2.07, 10.8] for Qwen3.6. Retail has only 40 tasks and is the lowest-powered benchmark. Its gains are positive for both actors (5.17 and 4.50 points), and the task-paired intervals include zero (lower bounds −0.130 and −0.290). The aligned runs also confirm the transfer you highlighted: the comparator, trained only on Qwen3 branches, improves Qwen3.6 on all three benchmarks (Section 4.3\color{blue}{\textbf{Section 4.3}}).

W2 & Q2: Consistent accounting for auxiliary models.

Thank you for this point. We agree that consistent accounting is the right standard for an efficiency claim. The submission charged nothing for the 0.5B models because OpenRouter lists comparably sized models as free. That tier is usage-capped, however, so your stricter accounting is more appropriate. The revision now charges every invoked model by its recorded tokens under one rule (Appendix B.8.2\color{blue}{\textbf{Appendix B.8.2}}, Table 9\color{blue}{\textbf{Table 9}} on p. 20). The comparator and candidate generator are billed at USD 0.01/M input and 0.02/M output tokens, including every comparator call used for gating and ranking, alongside the actor, replanning calls, and AgentRM's reward model.

COTA's advantage holds under this accounting (Table 1\color{blue}{\textbf{Table 1}} on p. 7, Section 4.3\color{blue}{\textbf{Section 4.3}}):

  • vs. AgentRM: Best-of-5 costs 2.41–3.36× as much as COTA, yet COTA scores higher in three of the four evaluated settings. Beam search costs 4.74–22.1× as much.
  • vs. Self-Reflection and Asym-AC: COTA is cheaper in 14 of 18 comparisons. In the remaining four, it costs at most USD 0.171 more per 100 episodes while scoring 2.50–25.0 points higher.
  • Runtime: COTA averages 1.39× Original on Qwen, vs. 4.21–5.66× for Best-of-5 and 8.71–21.6× for beam search.

We hope these results resolve both concerns: the gains hold under aligned multi-seed evaluation with paired uncertainty, and the efficiency advantage holds under fully consistent cost accounting. If so, we would be grateful if you would consider raising your score, and we are glad to answer any further questions.

Thank you authors for the additional experiements + results. It helps address my previous concerns on inconsistent baseline and accounting.

I have another question on demonstrated advance of COTA over prior work. While the shared runtime setup in Appendix B.7 and the ablations using the same backbone and branch data in Table 2 strengthen the empirical comparison. My remaining question concerns the main advance beyond Zhai et al.’s preference-trained evaluator and Asym-AC’s runtime advice to a frozen actor. Could you clarify which part of COTA provides the key new contribution, and which existing results best demonstrate its benefit over these approaches?

AuthorsOpenReview ↗

Response to Reviewer TYYP

Thank you for checking the new results. We are glad they address your concerns about the baselines and cost accounting. Your new question goes to the core of the paper, and we answer it below.

What is new: the auxiliary model only compares, and its comparisons drive the entire intervention.

COTA reduces the role of the runtime auxiliary model to a single primitive: judging, under a shared prefix, whether a sampled legal alternative beats the actor's proposal. These pairwise judgments decide both when to intervene, through the learned gate gθg_\theta (Eq. 10), which fires when at least RR of the KK alternatives win, and what to advise, namely the top-ranked winner. The actor then replans and keeps the final decision. Both prior methods assign the auxiliary model a harder role:

Zhai et al. [1]Asym-AC [2]COTA
What the auxiliary model judgesEach action separately, as a scalar (an implicit QQ-value learned by step-level DPO)The proposal in isolation, in absolute termsThe proposal and an alternative jointly, as a same-prefix pair
What it producesA score per actionFree-form corrective feedbackPairwise judgments, aggregated into the R/KR/K gate, and a concrete legal action as advice
Who chooses the executed actionThe evaluator (the highest-valued action)The actor, after the critiqueThe actor, through non-binding replanning

This narrower role is what makes a 0.5B model an effective advisor, and it is the practical property you highlighted in your review: a small comparator guides a larger agent without replacing its decision process.

Which results show the advantage. COTA differs from each prior method in one place, and one experiment isolates each difference.

  1. Against Zhai et al.: the comparison primitive. In the same-interface baselines (Appendix B.7\color{blue}{\textbf{Appendix B.7}}, Table 1\color{blue}{\textbf{Table 1}} on p. 7), the evaluator of Zhai et al. scores the same KK candidates inside COTA's gate, advice, and replanning, so only the learned scorer differs. COTA exceeds it on WebShop by 12.2 points with Qwen3 (53.2 vs. 41.0) and 10.3 points with Qwen3.6 (66.9 vs. 56.6), at less than half its cost and at 1.11–1.68× Original runtime versus 12.3–12.8×. Table 2\color{blue}{\textbf{Table 2}} (p. 8) then isolates the objective on the same 0.5B backbone and the same branch data: with constructive advice, pairwise comparison beats per-action absolute-QQ scoring in all six settings, by up to 9.70 points (Qwen3.6 ALFWorld: 94.0 vs. 84.3). This extends the ALFWorld result you noted in your review.

  2. Against Asym-AC: comparison-driven intervention. COTA outperforms Asym-AC in all nine actor–environment settings by 2.50–44.6 points (Table 1\color{blue}{\textbf{Table 1}}, Section 4.3\color{blue}{\textbf{Section 4.3}}), although Asym-AC's Qwen3-8B critic is 16× larger than COTA's comparator. The largest gap shows why: on Qwen3 WebShop, the free-form critic must decide in absolute terms whether a purchase is correct; it rejects about four in five purchase attempts, and Asym-AC falls to 8.58, against 37.3 for Original (see our response to Reviewer mgUP, W2). Table 15\color{blue}{\textbf{Table 15}} (Appendix C.5\color{blue}{\textbf{Appendix C.5}}, p. 22) quantifies what comparison-driven intervention contributes on matched WebShop seeds. With random advice, replacing a random gate by gθg_\theta raises the reward from 34.7 to 50.0 at nearly the same firing rate, and comparator-selected advice raises it further to 54.7.

The same property explains the transfer you noted: because the comparator judges only local differences, it was trained on Qwen3 branches and still improves Qwen3.6 and DeepSeek on all three benchmarks without retraining (Section 4.3\color{blue}{\textbf{Section 4.3}}).

In short, COTA's contribution is not a better evaluator or a better critic. It is a framework designed together with a comparison-only model: pairwise comparison, the R/KR/K gate, and non-binding replanning act together, so that a small model can efficiently steer a much larger one. We hope this clarifies the contribution. If it does, together with the resolved concerns about baselines and cost accounting, we would be grateful if you would consider raising your score. We are glad to answer any further questions.

[1] Zhai et al. Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models. AAAI 2025.

[2] Jiang et al. Asymmetric Actor-Critic for Multi-turn LLM Agents. arXiv:2604.00304, 2026.

AuthorsOpenReview ↗

Looking forward to further feedback

Dear Reviewer TYYP,

We hope this message finds you well.

Thank you again for engaging with our work so constructively, and for confirming that our additional experiments resolved your concerns about the baselines and cost accounting. We have since responded to your question on COTA's advance over Zhai et al. and Asym-AC. In short, the auxiliary model only compares: its pairwise judgments decide both when to intervene and what to advise, while the actor keeps the final decision, and each difference from prior work is isolated by a matched experiment (Tables 1, 2, and 15\color{blue}{\textbf{Tables 1, 2, and 15}}). As further evidence on your question, under the same rate-matched forced protocol as Table 2\color{blue}{\textbf{Table 2}}, the evaluator of Zhai et al. scores 26.1 on Qwen3 WebShop, against 46.7 for our comparator (see our latest response to Reviewer 4z7N).

As the discussion period is drawing to a close, we would appreciate knowing whether our response addresses your question. If it does, we would be grateful if you would consider updating your score. If anything remains unclear, please let us know, and we would welcome the chance to address it.

Thank you for your time and thoughtful feedback.

Best regards, The Authors

Reviewer_mgUPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Comparison only Tiny advisor (COTA) which acts as an advisor for frozen LLM agent during runtime. The article uses a small LM (0.5b model) trained on various actions at the same state comparing actor's proposal with K executable alternatives - using same prefix counterfactual branches. The intuition is that one doesn't need to know if an action is globally optimal or how to solve the task to spot a locally weak decision. COTA improves the utility in all the acts and benchmark settings i.e WebShop, τ3\tau^3 -Retail and ALFWorld. The contribution is also in lowering the cost as compared to methods like AgentRM.

Critical strengths
  • Exihibiting the strength of a SLM (i.e Qwen2.5-0.5B) as advisor for local comparisons is quite exceptional, given the gains it provides.
  • The paper experiments spans multiple datasets and different actors, and also includes learning objective and intervention ablations. The paper also presents a cross-actor preference analysis.
  • Consideration of tie-class and bidirectional check is sensible ways to limit unstable advice.
  • Use of same-prefix counterfactual branching is a clean approach to isolate long-horizon localized effective of action with cross-trajectory confounding.
Critical weaknesses
  • Discrepancy between table 1 and 3? Table 3 gives B=0 values of 36.6 and 44.2 for Webshop and retail, respectively, while Table 1 gives original values of 39.6 and 37.5. By definition, setting B=0 means there is zero intervention which means it is identical to original unassisted actor. While these values match for ALFworld, they diverge for WebShop and τ3\tau^3 -retail. There is no explanation why this differs, is it because of differences in random seeds, task subsets, sampling parameters for decoding etc.

  • Limited baseline configurations - Appendix B.7 uses zero-shot qwen3-8b critic for asym-AC. Because Asym-AC paper trains its critic, evaluating only zero-shot seems to be limited adaptation. Also, asym-AC's drop in WebShop score (10.1 vs. Original's 39.6 on Qwen3-8B) needs diagnosis. Additionally, agentRM is tested only with W1 = W2 = K = 4 beam search, however AgentRM also presents best-of-n sampling that smaller quality-cost trade off.

  • Zhai et al. (AAAI 2025) needs to be cited and compared. They train a lightweight (1.3B) step-level Q-value model from MCTS-derived action preferences and reuse it across LLM actor on WebShop by selecting the highest-scoring candidate action at inference.

  • Table 2 confounds intervention mechanism with intervention frequency. In table 12, for the absolute-Q model on ALFWorld,it shows that "Forced" uses an argmax 83.6% rate, while "Constructive/Selective" is at only 50.5%. As they trigger at different rates, it is not clear if the performance boost comes from letting the actor replan or simply from intervening less often. Also, the paper is missing the step-level intervention rates for the main pairwise method.

  • Section 3.2 says that a revised proposal is reviewed again while figure 1 shows that deepseek keeps its original proposal in 49.6%- 63.6% of replans. The stopping rule needs clarification for same-state review loop.

Questions for authors
  • Were table 1-original and table 3 b=0 runs were with the same tasks, seeds, prompts and decoding params? what caused the webshop and retail numbers to be different?
  • how many independent continuation M were used per action to form training labels and what is the webshop tie margin y?
  • Does table 5's retail physical-pair count exclude its test split?
  • How often did you hit fewer than k=4 distinct legal alternatives and what happens when the actor repeats its proposal after advice?
8 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/3]

Thank you for the thorough review, and for noting that a 0.5B advisor delivering these gains is "quite exceptional" and that same-prefix branching cleanly isolates local action effects. Your comments led to a revision that now gives direct evidence for COTA's central claim: the intervention role can be reduced to a local comparison that a tiny model learns reliably. Small value critics placed in the same interface trail COTA by 10.2–26.6 points, and Retrospex-IQL even falls below the original actor, while COTA improves it in every setting (W3), and rate-controlled ablations isolate the gate and the advice (W4). Below we respond to the Weaknesses (W) and Questions (Q).

W1 & Q1: Discrepancy between Tables 1 and 3\color{blue}{\textbf{Tables 1 and 3}}.

Thank you for catching this. The difference came only from rollout seeds: the WebShop and Retail B=0,1,2B=0,1,2 runs used a different seed from Table 1\color{blue}{\textbf{Table 1}}, while Full reused the main-table run. Tasks, prompts, and decoding parameters were unchanged. All Qwen results in Tables 1 and 3\color{blue}{\textbf{Tables 1 and 3}}, except the single-run AgentRM baselines, are now five-seed means ± SD, and Original/B=0B=0 and COTA/Full are identical across Tables 1 and 3\color{blue}{\textbf{Tables 1 and 3}} (pp. 7 and 9). The gains hold in all six Qwen settings: 15.9/6.72/5.17 points (Qwen3) and 10.4/6.42/4.50 points (Qwen3.6) on WebShop/ALFWorld/Retail (Section 4.3\color{blue}{\textbf{Section 4.3}}). As reported in Appendix C.6\color{blue}{\textbf{Appendix C.6}} (Table 16\color{blue}{\textbf{Table 16}} on p. 23), their 95% confidence intervals exclude zero in five of six settings across seeds, and on WebShop and ALFWorld for both actors when paired by task.

W2: Baseline configurations (Asym-AC, AgentRM).

  • Asym-AC. We use the published non-SFT variant with an off-the-shelf Qwen3-8B critic, since no author-trained SFT critic checkpoint is available. Appendix B.7\color{blue}{\textbf{Appendix B.7}} now states this explicitly. Note that Asym-AC-SFT requires a task-trained critic, and avoiding the cost of a task-capable critic is exactly the motivation for COTA.

    We traced the WebShop drop (8.58 vs. 37.3) across all five Qwen3 seeds: the critic blocks purchases that would have succeeded, and the actor obeys it until the episode runs out of steps. In this setting, the critic reviews only click[buy now] and rejects a purchase unless every attribute in the instruction is visible on the product page. Attributes such as "classic fit" or "machine wash" are rarely shown, so the critic rejects about four in five purchase attempts (e.g., 654 of 816 on one seed). It does so even though its prompt explicitly states that missing evidence is not evidence of a violated constraint. The Qwen3 actor follows each rejection, re-inspects the item or returns to search, and is rejected again, up to nine times in one episode, until it hits the 20-step limit. In 94–110 tasks per seed, the blocked purchase is exactly the one Original made after identical steps, earning 0.71–0.76 on WebShop's [0,1][0,1] reward. On these tasks, Asym-AC earns only 0.22–0.28. For example, on a task requesting a 60 ft HDMI cable, Original bought the right item for full reward, while the critic blocked the same purchase, claiming that the title said 50 ft.

    This failure is inherent to the method rather than to our configuration. With the same critic and code, Qwen3.6 sees 62–64% of its purchases blocked, but the stronger actor usually recovers and completes the purchase, so Asym-AC falls less far there (41.9 vs. Original's 56.4, Table 1\color{blue}{\textbf{Table 1}}). The contrast also illustrates COTA's motivation. A free-form critic must judge in absolute terms whether a proposal is correct, and an off-the-shelf model does so unreliably. COTA never judges a proposal in isolation: its comparator only judges whether each of KK sampled legal alternatives beats the proposal, the gate fires only when at least RR of them do, and its advice is non-binding with at most two replans per step, so it cannot hold up a purchase for more than two replans per step.

  • AgentRM Best-of-5. Following your suggestion, we add Best-of-5 next to beam search in Table 1\color{blue}{\textbf{Table 1}} (p. 7). It is cheaper than beam search but still costs 2.41–3.36× as much as COTA, and COTA scores higher in three of the four evaluated settings (e.g., Qwen3 WebShop: 53.2 vs. 46.4).

AuthorsOpenReview ↗

Rebuttal by Authors [2/3]

W3: Comparison with Zhai et al. [1].

Thank you for the pointer. We now discuss this work in Section 5\color{blue}{\textbf{Section 5}} and add its critic [1], together with the related Retrospex-IQL critic [2], as baselines in Table 1\color{blue}{\textbf{Table 1}} (p. 7), with implementation details in Appendix B.7\color{blue}{\textbf{Appendix B.7}}. Each scores the same KK candidates inside COTA's gate-and-advice interface, so only the learned scorer differs. Retrospex-IQL falls below the original actor in all four evaluated settings, and COTA exceeds it by 12.5–26.6 points at lower cost and runtime. The critic of Zhai et al. [1], run on WebShop (the only benchmark with released training data), improves Original by just 3.70 points for Qwen3 and is on par for Qwen3.6 (41.0 and 56.6 vs. COTA's 53.2 and 66.9), at over twice COTA's cost and 12.3–12.8× Original runtime (COTA: 1.11–1.68×). This is the distinction your review asked us to sharpen: both baselines give each action its own scalar value (Retrospex-IQL by TD regression of absolute QQ-values, [1] through an implicit QQ-value learned by step-level DPO), whereas COTA's comparator judges the proposal and each alternative jointly under the shared prefix. The shared prefix cancels, leaving a local sign that a 0.5B model learns reliably. Our absolute-QQ vs. pairwise ablation on the same backbone and data (Table 2\color{blue}{\textbf{Table 2}}) isolates the objective: pairwise comparison wins in all six constructive settings.

[1] Zhai et al. Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models. AAAI 2025.

[2] Xiang et al. Retrospex: Language Agent Meets Offline Reinforcement Learning Critic. EMNLP 2024.

W4: Intervention frequency confounds Table 2\color{blue}{\textbf{Table 2}}. Missing step-level rates.

We appreciate this insightful comment. It changed how we run the ablation. In the revised Table 2\color{blue}{\textbf{Table 2}} (p. 8), forced execution uses a random gate calibrated to COTA's step-level first-review rate. We apply the same frequency control to absolute-QQ constructive intervention, so the two objectives are also compared at equal frequency under constructive advice. This matters most on ALFWorld and Retail, where absolute value regression is poorly suited to binary task outcomes and the absolute-QQ gate would otherwise fire far more often than COTA. Table 12\color{blue}{\textbf{Table 12}} (p. 21) reports that rate, counting only the first check at each decision state: 36.9/49.7% on WebShop, 33.7/19.2% on ALFWorld, and 7.45/4.81% on Retail (Qwen3/Qwen3.6). With frequency controlled, the conclusions in Section 4.4\color{blue}{\textbf{Section 4.4}} are sharper:

  • The pairwise objective wins on its own merits. At matched rates, pairwise Forced beats absolute-QQ Forced in five of six settings and ties in the sixth (e.g., Qwen3 ALFWorld: 76.1 vs. 61.2). With constructive advice, it exceeds absolute-QQ in all six settings (e.g., Qwen3 ALFWorld: 90.3 vs. 82.1).
  • Constructive advice wins on its own merits. At the same rate, forced execution falls below Original on ALFWorld and Retail, whereas the same comparator used as advice raises Qwen3 to 90.3 and 45.0, respectively.
  • The gate and the advice each matter. On matched WebShop seeds (Appendix C.5\color{blue}{\textbf{Appendix C.5}}, Table 15\color{blue}{\textbf{Table 15}} on p. 22), random gating with random advice scores 34.7, below Original (37.2). The learned gate gθg_\theta lifts this to 50.0 with the same random advice, and comparator-selected advice reaches 54.7 at a virtually identical firing rate (36.8% vs. 36.9%). Neither replanning without the comparator nor the comparator's choice executed directly suffices. COTA's gains come from the comparator, the R/KR/K gate, and non-binding replanning acting together.
AuthorsOpenReview ↗

Rebuttal by Authors [3/3]

W5 & Q4: Review-loop stopping and candidate availability.

Thank you for flagging this missing detail, which we now clarify in Section 4.2\color{blue}{\textbf{Section 4.2}} and the Figure 1\color{blue}{\textbf{Figure 1}} caption, consistent with Section 3.2\color{blue}{\textbf{Section 3.2}}. Each environment step permits at most two replans. The loop ends when the gate accepts the current proposal or the budget is exhausted, and the latest proposal is then executed. An unchanged proposal is reviewed again, so "Keep" does not terminate the loop. Figure 1\color{blue}{\textbf{Figure 1}} counts individual replan events: "Keep" means the proposal matches the one reviewed in that round, and one step can contribute two events.

As reported in Appendix B.6\color{blue}{\textbf{Appendix B.6}} (p. 18), fewer than four distinct legal alternatives occur in 22.0% of WebShop rounds, 0% of ALFWorld rounds, and 38.0% of Retail rounds, including empty sets. In such rounds, COTA compares the proposal only against the available alternatives, and with none available it executes the proposal unchanged. This reflects the conservative design: COTA never invents actions, which is consistent with Retail's low intervention rate (7.45%) and its smaller but positive gains.

Q2: Continuations and tie margin.

We use M=4M=4 independent continuations per action under the fixed source actor. On WebShop's [0,1][0,1] reward, pairs whose mean returns differ by at most γ=0.10\gamma=0.10 are labeled ties, while ALFWorld and Retail use γ=0\gamma=0 on the averaged binary outcomes. Section 4.2\color{blue}{\textbf{Section 4.2}} and Appendix B.4\color{blue}{\textbf{Appendix B.4}} (p. 17) now state these settings.

Q3: Retail physical-pair count.

Thank you for the careful check. It is a typo. The corrected total in Table 5\color{blue}{\textbf{Table 5}} (Appendix B.3\color{blue}{\textbf{Appendix B.3}}, p. 17) is 5,736 physical pairs.

We hope these revisions resolve your concerns, and that the new same-interface baselines and controls show what COTA contributes beyond existing critics: a tiny model becomes an effective advisor once its role is reduced to local comparison. If so, we would be grateful if you would consider raising your score, and we are glad to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Paper Revision

We thank all reviewers for their constructive feedback, and we have responded to each reviewer individually. We have also uploaded a Paper Revision including additional baselines, controls, statistical analyses, and implementation details. Revised text and the captions of new or revised tables are highlighted in blue.

  • Section 3.4\color{blue}{\textbf{Section 3.4}} (Pages 5): We clarify that R/KR/K is a threshold on the empirical domination estimate and that Proposition 3.1 is an error decomposition.
  • Section 4.2\color{blue}{\textbf{Section 4.2}} (Page 6): We specify the review-loop stopping rule, the number of continuations, and the tie margin.
  • Table 1\color{blue}{\textbf{Table 1}} (Page 7): We report five-seed means ± SD for all Qwen methods except AgentRM and add three baselines: AgentRM Best-of-5, Zhai et al. [1], and Retrospex-IQL [2]. All costs now include the comparator and candidate generator.
  • Table 2\color{blue}{\textbf{Table 2}} (Page 8): We calibrate forced execution and absolute-QQ constructive intervention to COTA's step-level intervention rate, and report that rate.
  • Figure 1\color{blue}{\textbf{Figure 1}} (Page 8): We clarify that each replan event is counted separately and that Keep is not a stopping condition.
  • Table 3\color{blue}{\textbf{Table 3}} (Page 9): We rerun the budget study with the same five seeds as Table 1\color{blue}{\textbf{Table 1}} and clarify the definition of the budget BB.
  • Section 5\color{blue}{\textbf{Section 5}} (Page 9): We add a discussion of small step-level value critics [1, 2].
  • Table 4\color{blue}{\textbf{Table 4}} (Page 16): We add the number of evaluation seeds per benchmark.
  • Table 5\color{blue}{\textbf{Table 5}} (Page 17): We correct the Retail physical-pair count.
  • Appendix B.6\color{blue}{\textbf{Appendix B.6}} (Page 18): We report how often fewer than KK legal alternatives are available and how COTA handles such rounds.
  • Appendix B.7\color{blue}{\textbf{Appendix B.7}} (Page 19): We add implementation details of the new baselines and the Asym-AC variant.
  • Table 9\color{blue}{\textbf{Table 9}} (Page 20): We add the price of the 0.5B comparator and candidate generator.
  • Table 10\color{blue}{\textbf{Table 10}} (Page 20): We add held-out agreement between comparator-selected winners and rollout preferences.
  • Table 12\color{blue}{\textbf{Table 12}} (Page 21): We add COTA's step-level intervention counts.
  • Table 15\color{blue}{\textbf{Table 15}} (Page 22): We add matched-seed controls that separate COTA's gate from its advice, including random gating with random advice.
  • Table 16\color{blue}{\textbf{Table 16}} (Page 23): We add seed-level and task-paired 95% confidence intervals and McNemar tests for COTA's gains.

[1] Zhai et al. Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models. AAAI 2025.

[2] Xiang et al. Retrospex: Language Agent Meets Offline Reinforcement Learning Critic. EMNLP 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.