- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SIPA, a training-free test-time RL framework that improves LLM agents' action selection using immediate environment feedback. While the actor generates candidate actions, a lightweight proposer executes alternative actions in isolated copies of the current environment. The resulting observations are evaluated by a critic and incorporated through APA, which combines the actor's action preferences with the critic's rejection preferences using a KL-regularized policy update. The theoretical analysis focuses on the update's stability under feedback errors and sufficient conditions for expected-return improvement. Experiments on five multi-turn benchmarks using Qwen3 models from 4B to 14B demonstrate improved task accuracy over existing test-time adaptation baselines, with relatively limited additional inference latency.
Critical strengths
- SIPA uses speculative execution to obtain immediate environment feedback and adapt action selection without parameter updates or historical experience. By overlapping speculative trials with actor reasoning, it adds only about 10% per-step latency. The framework is straightforward and provides a practical alternative to experience-based test-time adaptation.
- The evaluation covers five interactive benchmarks, multiple TTA baselines, and Qwen3 models from 4B to 14B. SIPA achieves the best or tied-best accuracy across all five benchmarks at 4B, improving average accuracy by 8.82 percentage points over the strongest baseline. The paper also includes comprehensive ablation studies and analyses of cases with limited improvements, strengthening the credibility of the results.
Critical weaknesses
- The theoretical results hold for arbitrary finite correction signals, so they mainly characterize generic properties of KL-regularized reweighting, not for the proposed asymmetric rejection feedback. In particular, the return-improvement guarantee in Thm. 3 is conditional on the critic corrections ordering all actor candidates consistently with . It is not clear why rejection preferences derived from single-step speculative observations should satisfy this condition, and the paper does not examine how often it holds in practice.
- The current baselines mainly demonstrate the benefits of incorporating speculative execution feedback compared to agents without such feedback. However, they do not sufficiently establish whether the proposed APA update is more effective than simpler alternatives for utilizing the same feedback.
Questions for authors
- Appendix D indicates that trial feedback includes termination status, and Fig. 3 shows the critic receiving done = true for the action that completes the task. Since termination is likely strongly correlated with success in BabyAI and ALFWorld, could the authors report results with termination status withheld from the critic?
- The sensitivity analysis uses all 219 ToolSandbox tasks, and the default configuration is the best point in every sweep. Was this configuration selected independently of the evaluation set? Did the baselines receive comparable hyperparameter tuning, and were they subject to the same thinking-token budget (1,024) and context window (16 messages) as the SIPA actor?
- SIPA performs up to six speculative trials per committed action, but the reported Steps metric counts committed actions only. Could the authors report total environment calls per task, or compare methods under matched interaction budgets?
5 comments
Response to Reviewer HEeY (Part 2)
W2. APA under matched access to trial feedback
Several baselines use execution feedback, as described in Appendix D (p. 28). Reflexion carries observations and reflections from one attempt into the next. SMURFS checks tool responses and backtracks when a plan fails. Critic-only is the closest comparison. At the current decision state, it executes the proposer’s candidates in separate branches, scores them using their observed outcomes, and samples from the critic’s distribution over the tested actions. However, it reaches only 23.71% mean accuracy on Qwen3-4B, compared with 48.64% for SIPA.
To isolate how the decision rule uses that feedback, we evaluated two ablations. Both retain SIPA’s parallel actor and proposer branches and the same trial protocol and budget. With In-context selection, the actor reads the raw trial outcomes alongside its original reasoning and selects from its candidates. With Failure filtering, it takes its highest-ranked candidate whose matching trial returned neither an execution error nor an explicit failure. If none qualifies, it takes its original top choice.
| Method · accuracy (%) | ToolSandbox | ScienceWorld | BFCL | ALFWorld | BabyAI | Mean |
|---|---|---|---|---|---|---|
| Actor-only | 15.98 | 16.02 | 19.17 | 76.12 | 57.89 | 37.04 |
| Reflexion | 15.53 | 20.31 | 25.83 | 72.39 | 31.58 | 33.13 |
| SMURFS | 14.16 | 2.34 | 2.92 | 27.61 | 52.63 | 19.93 |
| Critic-only | 5.48 | 3.91 | 6.67 | 20.90 | 81.58 | 23.71 |
| In-context selection | 16.89 | 18.75 | 25.83 | 82.84 | 78.95 | 44.65 |
| Failure filtering | 17.35 | 17.19 | 23.75 | 79.85 | 68.42 | 41.31 |
| SIPA | 18.72 | 20.31 | 30.00 | 87.31 | 86.84 | 48.64 |
Both controls improved on Actor-only. SIPA outperformed both on every benchmark, by an average of 3.99 percentage points over In-context selection and 7.33 over Failure filtering. In In-context selection, the actor kept its original action in 54.7% of the decision steps where its initial choice was wrong, even after seeing the trial outcomes. Persistence after disconfirming evidence has also been observed in controlled studies of belief revision [2]. APA can downweight an action that executes successfully but makes little progress, while preserving the actor’s relative preferences among candidates that receive the same correction. Under matched trial access, this graded use of feedback yielded better decisions than either direct reconsideration or a binary filter. These results are reported in the revised manuscript (Table 1, p. 6, and Appendix C.1, Table 6, p. 18).
Q1. Accuracy with termination status withheld
Termination status is part of the environment’s public response, and all four baselines in the table have access to it. As described in Appendix D (p. 27), the critic receives observable execution feedback and has no access to benchmark scores or success labels derived from them.
For this response, we measured the contribution of this field by removing the explicit termination status from the critic’s trial observations on ALFWorld and BabyAI. We retained the ordinary observation text, the environment’s stopping rules, and the original scoring procedure. The full-access results are from Table 1 (p. 6).
| Method | Access to termination status | ALFWorld accuracy (%) | BabyAI accuracy (%) |
|---|---|---|---|
| Actor-only | ✓ | 76.12 | 57.89 |
| Critic-only | ✓ | 20.90 | 81.58 |
| Reflexion | ✓ | 72.39 | 31.58 |
| SMURFS | ✓ | 27.61 | 52.63 |
| SIPA | ✓ | 87.31 | 86.84 |
| SIPA without termination status | ✗ | 84.33 | 84.21 |
| Gain without termination over Actor-only (pp) | — | +8.21 | +26.32 |
Access to termination status increased SIPA’s accuracy by 2.98 percentage points on ALFWorld and 2.63 on BabyAI. Even with the flag withheld, SIPA led the strongest baseline on each benchmark by 8.21 points over Actor-only on ALFWorld and 2.63 points over Critic-only on BabyAI. While all four baselines retained access to the termination flag, SIPA still outperformed all four on both benchmarks with the flag removed from its own trial feedback.
Response to Reviewer HEeY (Part 3)
Q2. Hyperparameter selection and baseline inference budgets
We apologize for the confusion about which ToolSandbox tasks were used for tuning. SIPA requires no weight training, but its update strength, KL cap, and candidate limit have no established defaults. We therefore used the ToolSandbox training split as a development set to choose these three parameters once. Appendix C.1 (p. 19) now specifies the selection split and the configuration used for evaluation. Figure 5 (p. 9) shows sensitivity to varying each parameter around that configuration.
We froze before evaluating the 219 held-out ToolSandbox tasks. With Qwen3-4B, SIPA solved 41/219 tasks (18.72%), compared with 35/219 (15.98%) for Actor-only (Table 1, p. 6). We carried that same configuration over to all four other benchmarks for the main comparison, with no benchmark-specific retuning.
For each baseline, we used the default method-specific hyperparameters from its official repository. Across all baselines and the SIPA actor, we matched the backbone model, the 1,024-token reasoning budget, and the history window of 16 recent messages. Appendix D describes the actor’s inference settings (pp. 26–27).
Q3. Environment calls and accuracy under matched budgets
Following the reviewer's suggestion, we counted the additional environment calls made by SIPA and four baselines on ToolSandbox. A committed call executes an action on the live trajectory, including a retry, and corresponds to one Step in Table 1 (p. 6). An additional call is an environment execution outside that trajectory, including a speculative trial or an attempt later discarded. We averaged each call column over the tasks solved by the corresponding method (Appendix C.4, Table 16, p. 26).
| Method | ToolSandbox accuracy (%) | Committed calls/successful task | Additional calls/successful task | Total calls/successful task |
|---|---|---|---|---|
| Actor-only | 15.98 | 24.91 | 0 | 24.91 |
| Critic-only | 5.48 | 27.00 | 67.50 | 94.50 |
| Reflexion | 15.53 | 24.38 | 50.00 | 74.38 |
| SMURFS | 14.16 | 23.97 | 61.13 | 85.10 |
| SIPA | 18.72 | 8.51 | 20.44 | 28.95 |
SIPA averaged 28.95 total calls per successful task, 4.04 more than Actor-only. Of the three baselines that also use trial feedback, the least costly was Reflexion at 74.38 calls per successful task.
Two features explain how SIPA limits this overhead. First, shorter committed trajectories require fewer rounds of speculation. SIPA finished successful tasks in 8.51 committed calls on average, saving 16.40 relative to Actor-only and offsetting most of its 20.44 additional calls. Second, caps the number of proposals in the critic branch. The proposer decides how many candidates to generate at each state. Across successful tasks, SIPA averaged only 2.40 additional calls per committed step.
Figure 9 (p. 25) reports accuracy under per-task limits on committed actions. To test whether the gain survives when trials count against the same budget, we evaluated SIPA and Actor-only under identical caps on total environment calls, including speculative trials executed in parallel (Appendix C.4, Table 17, p. 26).
| Total calls allowed per task | Actor-only accuracy (%) | SIPA accuracy (%) | SIPA − Actor-only (pp) |
|---|---|---|---|
| 30 | 13.24 | 11.87 | −1.37 |
| 50 | 14.61 | 15.98 | +1.37 |
| 100 | 15.98 | 17.81 | +1.83 |
With 30 calls per task, Actor-only led by 1.37 percentage points. At 50 and 100 calls, SIPA led by 1.37 and 1.83 points, respectively. Under the two larger budgets, spending some of the same call allowance on trials improved the chance of completing the task.
This trade-off is relevant when a trial is cheap relative to model inference, as in agentic search over a local index, read-only queries to a product catalog, or testing moves in a lightweight game simulator. In these settings, accuracy and elapsed time can matter more to a user than the number of calls. SIPA overlaps trial execution with actor reasoning (Fig. 4a, p. 8). On the same single-GPU setup, mean latency per decision rose by 1.21 seconds, from 11.81 to 13.02 seconds (Table 3, p. 7). With this 10.2% increase in per-step latency, mean task accuracy across the five benchmarks improved by 11.60 percentage points, from 37.04% to 48.64% (Table 1, p. 6).
References
[1] Li, Y., Lin, Z., Deng, A., Zhang, X., He, Y., Ji, S., Cao, T., & Hooi, B. (2026). Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306, 68204–68230.
[2] Sadhu, J., Shahad, S., & Marino, K. (2026). DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pp. 36952–36985.
Response to Reviewer HEeY (Part 1)
Thank you for your constructive feedback and for the specific comparisons you suggested. We address each of your comments in turn below.
W1. Rejection feedback and long-term action value
We agree that the KL analysis applies to a range of correction signals. The evidence for choosing rejection feedback comes from the probe comparison in Table 2 (p. 7). With the actor prior and trial protocol held fixed, asking the critic which action to avoid yielded 48.64% mean task accuracy. Asking which action to take yielded 44.22%.
Theorem 3 (p. 5) helps explain why rejection can work with only partial information about long-term return. Agreement with Qπ0 on every pair of actor candidates is sufficient but not necessary to guarantee that the update preserves or improves expected return at that state. Equation (19) in Appendix B.3 (p. 16) also explains how an update can help when some pairs are misranked. For instance, uniformly downweighting a group of actions improves expected return whenever their actor-weighted mean continuation value is lower than that of the remaining candidates. A trial can therefore provide enough evidence to reject an action even if it cannot identify the best alternative.
To test this implication, we report an additional diagnostic here. We compared the probes at the same decision states, holding the actor prior, candidates, trial observations, and APA hyperparameters fixed. Following the estimator in Section 4.2 of JitRL [1], we estimated Qπ0 from the returns of matching actions at similar states in separate Actor-only trajectories. For each probe j, we measured the change in estimated continuation value from the actor prior p to the updated distribution qj.
\qquad j\in\{\mathrm{neg},\mathrm{pos}\}.$$ We averaged both gains over states with historical support for every action whose probability changed under either probe. The last column subtracts the positive probe gain from the negative probe gain on those same states. |Benchmark|Negative probe gain|Positive probe gain|Paired difference [95% CI]| |-|-:|-:|-:| |BabyAI|+0.057|+0.032|**+0.025 [−0.003, +0.056]**| |ALFWorld|+0.033|+0.021|**+0.012 [+0.003, +0.022]**| |ToolSandbox|+0.013|−0.004|**+0.017 [+0.002, +0.034]**| |ScienceWorld|+0.022|+0.012|**+0.010 [−0.002, +0.023]**| |BFCL|+0.016|+0.011|**+0.005 [−0.004, +0.015]**| The negative probe had a positive estimated gain and a larger gain than the positive probe on all five benchmarks. On the states covered by the diagnostic, rejection feedback thus shifted probability toward actions with higher estimated continuation value on average. Together with Table 2, this supports **using a trial to identify which actions to downweight**, even when it leaves the best alternative unresolved.