acceptodds
Under review as a conference paper at ICLR 2027

SIPA: Test-Time Reinforcement Learning through Speculative Policy Adaptation

Abstract

Reinforcement learning (RL) has become a central paradigm for language-model post-training, yet its gains do not always transfer to unfamiliar deployment environments. Experience-based test-time adaptation has emerged as a promising approach, drawing on accumulated experience to guide subsequent decisions. However, collecting relevant experience is computationally demanding, and the resulting guidance may be inaccurate or poorly matched to the current state. We introduce Speculative Inference-Time Policy Adaptation (SIPA), a test-time RL framework that adapts the current action distribution using immediate execution feedback, without parameter updates or pre-collected trajectories. While the actor reasons, a lightweight branch proposes and tests candidate actions in isolated copies of the current environment state. Asymmetric Policy Adaptation (APA) combines the actor's selection preferences with the critic's execution-based rejection feedback. A KL-regularized logit update incorporates these signals before action commitment while bounding deviation from the actor's prior. We establish stability bounds under feedback error and sufficient conditions for expected-return improvement. Across five multi-turn benchmarks, SIPA achieves a 22.1% relative improvement in average accuracy over the strongest baseline using only 69.6% as many committed steps.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-time reinforcement learning, language agents, policy adaptation, speculative execution, environment feedback

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_HEeYOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SIPA, a training-free test-time RL framework that improves LLM agents' action selection using immediate environment feedback. While the actor generates candidate actions, a lightweight proposer executes alternative actions in isolated copies of the current environment. The resulting observations are evaluated by a critic and incorporated through APA, which combines the actor's action preferences with the critic's rejection preferences using a KL-regularized policy update. The theoretical analysis focuses on the update's stability under feedback errors and sufficient conditions for expected-return improvement. Experiments on five multi-turn benchmarks using Qwen3 models from 4B to 14B demonstrate improved task accuracy over existing test-time adaptation baselines, with relatively limited additional inference latency.

Critical strengths
  1. SIPA uses speculative execution to obtain immediate environment feedback and adapt action selection without parameter updates or historical experience. By overlapping speculative trials with actor reasoning, it adds only about 10% per-step latency. The framework is straightforward and provides a practical alternative to experience-based test-time adaptation.
  2. The evaluation covers five interactive benchmarks, multiple TTA baselines, and Qwen3 models from 4B to 14B. SIPA achieves the best or tied-best accuracy across all five benchmarks at 4B, improving average accuracy by 8.82 percentage points over the strongest baseline. The paper also includes comprehensive ablation studies and analyses of cases with limited improvements, strengthening the credibility of the results.
Critical weaknesses
  1. The theoretical results hold for arbitrary finite correction signals, so they mainly characterize generic properties of KL-regularized reweighting, not for the proposed asymmetric rejection feedback. In particular, the return-improvement guarantee in Thm. 3 is conditional on the critic corrections ordering all actor candidates consistently with Qπ0Q^{\pi_0}. It is not clear why rejection preferences derived from single-step speculative observations should satisfy this condition, and the paper does not examine how often it holds in practice.
  2. The current baselines mainly demonstrate the benefits of incorporating speculative execution feedback compared to agents without such feedback. However, they do not sufficiently establish whether the proposed APA update is more effective than simpler alternatives for utilizing the same feedback.
Questions for authors
  1. Appendix D indicates that trial feedback includes termination status, and Fig. 3 shows the critic receiving done = true for the action that completes the task. Since termination is likely strongly correlated with success in BabyAI and ALFWorld, could the authors report results with termination status withheld from the critic?
  2. The sensitivity analysis uses all 219 ToolSandbox tasks, and the default configuration is the best point in every sweep. Was this configuration selected independently of the evaluation set? Did the baselines receive comparable hyperparameter tuning, and were they subject to the same thinking-token budget (1,024) and context window (16 messages) as the SIPA actor?
  3. SIPA performs up to six speculative trials per committed action, but the reported Steps metric counts committed actions only. Could the authors report total environment calls per task, or compare methods under matched interaction budgets?
5 comments
AuthorsOpenReview ↗

Response to Reviewer HEeY (Part 1)

Thank you for your constructive feedback and for the specific comparisons you suggested. We address each of your comments in turn below.

W1. Rejection feedback and long-term action value

We agree that the KL analysis applies to a range of correction signals. The evidence for choosing rejection feedback comes from the probe comparison in Table 2 (p. 7). With the actor prior and trial protocol held fixed, asking the critic which action to avoid yielded 48.64% mean task accuracy. Asking which action to take yielded 44.22%.

Theorem 3 (p. 5) helps explain why rejection can work with only partial information about long-term return. Agreement with Qπ0Q^{\pi_0} on every pair of actor candidates is sufficient but not necessary to guarantee that the update preserves or improves expected return at that state. Equation (19) in Appendix B.3 (p. 16) also explains how an update can help when some pairs are misranked. For instance, uniformly downweighting a group of actions improves expected return whenever their actor-weighted mean continuation value is lower than that of the remaining candidates. A trial can therefore provide enough evidence to reject an action even if it cannot identify the best alternative.

To test this implication, we report an additional diagnostic here. We compared the probes at the same decision states, holding the actor prior, candidates, trial observations, and APA hyperparameters fixed. Following the estimator in Section 4.2 of JitRL [1], we estimated Qπ0Q^{\pi_0} from the returns of matching actions at similar states in separate Actor-only trajectories. For each probe jj, we measured the change in estimated continuation value from the actor prior pp to the updated distribution qjq_j.

\qquad j\in\{\mathrm{neg},\mathrm{pos}\}.$$ We averaged both gains over states with historical support for every action whose probability changed under either probe. The last column subtracts the positive probe gain from the negative probe gain on those same states. |Benchmark|Negative probe gain|Positive probe gain|Paired difference [95% CI]| |-|-:|-:|-:| |BabyAI|+0.057|+0.032|**+0.025 [−0.003, +0.056]**| |ALFWorld|+0.033|+0.021|**+0.012 [+0.003, +0.022]**| |ToolSandbox|+0.013|−0.004|**+0.017 [+0.002, +0.034]**| |ScienceWorld|+0.022|+0.012|**+0.010 [−0.002, +0.023]**| |BFCL|+0.016|+0.011|**+0.005 [−0.004, +0.015]**| The negative probe had a positive estimated gain and a larger gain than the positive probe on all five benchmarks. On the states covered by the diagnostic, rejection feedback thus shifted probability toward actions with higher estimated continuation value on average. Together with Table 2, this supports **using a trial to identify which actions to downweight**, even when it leaves the best alternative unresolved.
AuthorsOpenReview ↗

Response to Reviewer HEeY (Part 2)

W2. APA under matched access to trial feedback

Several baselines use execution feedback, as described in Appendix D (p. 28). Reflexion carries observations and reflections from one attempt into the next. SMURFS checks tool responses and backtracks when a plan fails. Critic-only is the closest comparison. At the current decision state, it executes the proposer’s candidates in separate branches, scores them using their observed outcomes, and samples from the critic’s distribution over the tested actions. However, it reaches only 23.71% mean accuracy on Qwen3-4B, compared with 48.64% for SIPA.

To isolate how the decision rule uses that feedback, we evaluated two ablations. Both retain SIPA’s parallel actor and proposer branches and the same trial protocol and budget. With In-context selection, the actor reads the raw trial outcomes alongside its original reasoning and selects from its candidates. With Failure filtering, it takes its highest-ranked candidate whose matching trial returned neither an execution error nor an explicit failure. If none qualifies, it takes its original top choice.

Method · accuracy (%)ToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only15.9816.0219.1776.1257.8937.04
Reflexion15.5320.3125.8372.3931.5833.13
SMURFS14.162.342.9227.6152.6319.93
Critic-only5.483.916.6720.9081.5823.71
In-context selection16.8918.7525.8382.8478.9544.65
Failure filtering17.3517.1923.7579.8568.4241.31
SIPA18.7220.3130.0087.3186.8448.64

Both controls improved on Actor-only. SIPA outperformed both on every benchmark, by an average of 3.99 percentage points over In-context selection and 7.33 over Failure filtering. In In-context selection, the actor kept its original action in 54.7% of the decision steps where its initial choice was wrong, even after seeing the trial outcomes. Persistence after disconfirming evidence has also been observed in controlled studies of belief revision [2]. APA can downweight an action that executes successfully but makes little progress, while preserving the actor’s relative preferences among candidates that receive the same correction. Under matched trial access, this graded use of feedback yielded better decisions than either direct reconsideration or a binary filter. These results are reported in the revised manuscript (Table 1, p. 6, and Appendix C.1, Table 6, p. 18).

Q1. Accuracy with termination status withheld

Termination status is part of the environment’s public response, and all four baselines in the table have access to it. As described in Appendix D (p. 27), the critic receives observable execution feedback and has no access to benchmark scores or success labels derived from them.

For this response, we measured the contribution of this field by removing the explicit termination status from the critic’s trial observations on ALFWorld and BabyAI. We retained the ordinary observation text, the environment’s stopping rules, and the original scoring procedure. The full-access results are from Table 1 (p. 6).

MethodAccess to termination statusALFWorld accuracy (%)BabyAI accuracy (%)
Actor-only✓76.1257.89
Critic-only✓20.9081.58
Reflexion✓72.3931.58
SMURFS✓27.6152.63
SIPA✓87.3186.84
SIPA without termination status✗84.3384.21
Gain without termination over Actor-only (pp)—+8.21+26.32

Access to termination status increased SIPA’s accuracy by 2.98 percentage points on ALFWorld and 2.63 on BabyAI. Even with the flag withheld, SIPA led the strongest baseline on each benchmark by 8.21 points over Actor-only on ALFWorld and 2.63 points over Critic-only on BabyAI. While all four baselines retained access to the termination flag, SIPA still outperformed all four on both benchmarks with the flag removed from its own trial feedback.

AuthorsOpenReview ↗

Response to Reviewer HEeY (Part 3)

Q2. Hyperparameter selection and baseline inference budgets

We apologize for the confusion about which ToolSandbox tasks were used for tuning. SIPA requires no weight training, but its update strength, KL cap, and candidate limit have no established defaults. We therefore used the ToolSandbox training split as a development set to choose these three parameters once. Appendix C.1 (p. 19) now specifies the selection split and the configuration used for evaluation. Figure 5 (p. 9) shows sensitivity to varying each parameter around that configuration.

We froze (η,κ,G)=(32,10,6)(\eta,\kappa,G)=(32,10,6) before evaluating the 219 held-out ToolSandbox tasks. With Qwen3-4B, SIPA solved 41/219 tasks (18.72%), compared with 35/219 (15.98%) for Actor-only (Table 1, p. 6). We carried that same configuration over to all four other benchmarks for the main comparison, with no benchmark-specific retuning.

For each baseline, we used the default method-specific hyperparameters from its official repository. Across all baselines and the SIPA actor, we matched the backbone model, the 1,024-token reasoning budget, and the history window of 16 recent messages. Appendix D describes the actor’s inference settings (pp. 26–27).

Q3. Environment calls and accuracy under matched budgets

Following the reviewer's suggestion, we counted the additional environment calls made by SIPA and four baselines on ToolSandbox. A committed call executes an action on the live trajectory, including a retry, and corresponds to one Step in Table 1 (p. 6). An additional call is an environment execution outside that trajectory, including a speculative trial or an attempt later discarded. We averaged each call column over the tasks solved by the corresponding method (Appendix C.4, Table 16, p. 26).

MethodToolSandbox accuracy (%)Committed calls/successful taskAdditional calls/successful taskTotal calls/successful task
Actor-only15.9824.91024.91
Critic-only5.4827.0067.5094.50
Reflexion15.5324.3850.0074.38
SMURFS14.1623.9761.1385.10
SIPA18.728.5120.4428.95

SIPA averaged 28.95 total calls per successful task, 4.04 more than Actor-only. Of the three baselines that also use trial feedback, the least costly was Reflexion at 74.38 calls per successful task.

Two features explain how SIPA limits this overhead. First, shorter committed trajectories require fewer rounds of speculation. SIPA finished successful tasks in 8.51 committed calls on average, saving 16.40 relative to Actor-only and offsetting most of its 20.44 additional calls. Second, G=6G=6 caps the number of proposals in the critic branch. The proposer decides how many candidates to generate at each state. Across successful tasks, SIPA averaged only 2.40 additional calls per committed step.

Figure 9 (p. 25) reports accuracy under per-task limits on committed actions. To test whether the gain survives when trials count against the same budget, we evaluated SIPA and Actor-only under identical caps on total environment calls, including speculative trials executed in parallel (Appendix C.4, Table 17, p. 26).

Total calls allowed per taskActor-only accuracy (%)SIPA accuracy (%)SIPA − Actor-only (pp)
3013.2411.87−1.37
5014.6115.98+1.37
10015.9817.81+1.83

With 30 calls per task, Actor-only led by 1.37 percentage points. At 50 and 100 calls, SIPA led by 1.37 and 1.83 points, respectively. Under the two larger budgets, spending some of the same call allowance on trials improved the chance of completing the task.

This trade-off is relevant when a trial is cheap relative to model inference, as in agentic search over a local index, read-only queries to a product catalog, or testing moves in a lightweight game simulator. In these settings, accuracy and elapsed time can matter more to a user than the number of calls. SIPA overlaps trial execution with actor reasoning (Fig. 4a, p. 8). On the same single-GPU setup, mean latency per decision rose by 1.21 seconds, from 11.81 to 13.02 seconds (Table 3, p. 7). With this 10.2% increase in per-step latency, mean task accuracy across the five benchmarks improved by 11.60 percentage points, from 37.04% to 48.64% (Table 1, p. 6).

References

[1] Li, Y., Lin, Z., Deng, A., Zhang, X., He, Y., Ji, S., Cao, T., & Hooi, B. (2026). Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306, 68204–68230.

[2] Sadhu, J., Shahad, S., & Marino, K. (2026). DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pp. 36952–36985.

Reviewer_hR3jOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a test-time method for large language model (LLM) agents. At each step, while the actor (the LLM policy) thinks, a proposer suggests up to 6 candidate actions, and each one is tried once in a copy of the current environment state. Meanwhile, a critic (the same frozen LLM) reads the trial results and scores how likely each candidate is wrong. The actor's action distribution is then shifted away from rejected actions, with a KL distance bound, so it does not move too far from the original, and the final action is sampled and executed. No weights are updated. They test their method on 5 benchmarks (ToolSandbox, ScienceWorld, BFCL, ALFWorld, BabyAI), with Qwen3-4B, plus 8B and 14B. Their experiments show some benefit of their method.

Critical strengths
  1. The control of the experiments is clean because Actor-only keeps everything the same (candidates, scoring, temperature, resampling) and only turns off the update. So the +11.60 average performance benefit comes from the adaptation step itself.

  2. The idea is simple and clear. For example, it just tries sampled candidate actions in a copy of the environment and lowers the probability of the ones that fail, that's litterally it. Also, it improves performance on several benchmarks at test time without any parameter updates.

Critical weaknesses
  1. Novelty is very limited, just combination of well known technique. Parallel execution during thought is borrowed from Speculative Actions [1], which the paper says inspired it. Also, trial execution in an environment copy is standard one-step lookahead (tree search), group rejection scores follow [2] and the exponential feedback update is standard relative-entropy policy search [3, 4]. I think the main new part is using trial outcomes to reshape, rather than just verify, candidate choices.

  2. The comparison is not matched in information, and the most obvious baseline is missing. SIPA tests candidate actions on a copy of the current state before acting, whereas the memory and reflection baselines (ReasoningBank, ACE, JitRL, Reflexion, SMURFS) cannot. The obvious control is a plain filter: pick the actor's top-ranked candidate that survived the trial. Without this, it is unclear whether the KL update adds anything over basic trial-and-error filtering. Critic-only does not answer this because it discards the actor's prior entirely.

  3. The experimental evidence is weak. The 4B results do not report confidence intervals. On ScienceWorld the Actor-only control drops from 16.02 at 4B to 4.30 at 8B and 5.86 at 14B, so the large 8B/14B gains there are measured against an unexpectedly weak control.

  4. Environment-call costs are hidden, and practical applicability is really question marked. The claim of "69.6% as many committed steps" leaves out the speculative trials (up to 6 extra environment calls per step), which are "counted separately" but never reported. Requiring a copyable environment limits deployment: the paper itself says it needs "reliable isolation of speculative execution," which is hard to provide in real systems with irreversible side effects (e.g., executing transactions or sending external messages) unless a faithful sandbox exists.

[1] N. Ye el al. Speculative Actions: A Lossless Framework for Faster AI Agents. ICLR 2026.

[2] C. Yao el al. Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends. ICLR 2026.

[3] J. Peters el al.. Relative Entropy Policy Search. AAAI 2010.

[4] A. Abdolmaleki el al.. Maximum a Posteriori Policy Optimisation. ICLR 2018.

Questions for authors
  1. How does SIPA compare with a plain filter baseline: the actor's top choice among the candidates that did not fail in the trial?

  2. Are Tables 1 and 4 single runs? Can you give seed variation or confidence intervals at 4B, where the headline comes from?

  3. How was the default config (eta=32, kappa=10, G=6) chosen? It wins every sensitivity sweep on ToolSandbox; was it tuned on test tasks?

  4. How many environment calls does SIPA use in total, including trials, compared with each baseline?

  5. Why does Actor-only drop so much from 4B to 8B/14B on ScienceWorld (16.02 -> 4.30 -> 5.86)?

  6. Where does SIPA apply when the environment cannot be copied or reset, and what happens when trials are not possible?

6 comments
AuthorsOpenReview ↗

Response to Reviewer hR3j (Part 1)

Thank you for the constructive feedback on SIPA’s contribution and the comparisons suggested. We address each point below.

W1. Reasoning depth and exploration breadth

An agent can improve a decision through reasoning depth and exploration breadth. Deeper reasoning helps the actor judge which action serves the long-term goal, but that judgment can rest on a mistaken prediction about what an action will do. Trying several actions from the same state tests those predictions, even when their immediate outcomes leave the best continuation unresolved. SIPA brings both sources of evidence to the same decision. While the actor deliberates, a parallel branch tests alternatives whose outcomes can change the action ultimately chosen. The lossless formulation of Speculative Actions [1] uses speculation to accelerate execution while preserving the original agent’s behavior. Section 5 (p. 9) now makes this distinction explicit.

APA exploits an asymmetry in what a trial reveals. Evidence against one action can justify lowering its probability while preserving the actor’s relative preferences among equally penalized alternatives. Group-relative REINFORCE [2] uses within-group rewards to estimate a training gradient. APA uses relative rejection scores to update the current action distribution under a relative-entropy objective related to REPS [3] and MPO [4]. Theorem 3 (p. 5) and Eq. (19) in Appendix B.3 (p. 16) identify when such partial evidence suffices for improvement. Even with an incomplete ranking, an update improves expected return at a state if, on balance, it shifts probability toward actions with higher continuation value under the original actor. Corollary 4 in Appendix B.3 (p. 17) connects this criterion to task performance by expressing the change in expected task return as the expected sum of these local gains along trajectories of the adapted policy.

W2 / Q1. Action selection under matched trial access

We would like to clarify how the baselines use execution feedback (Appendix D, p. 28). Reflexion uses one attempt to inform the next, and SMURFS uses tool responses to verify steps and backtrack. Critic-only tries the proposer’s candidates from the current state and samples from its scores of their outcomes. Since it discards the actor’s ranking, it leaves open how much APA adds when a filter has access to both the ranking and the trials.

We added the suggested Failure filtering control and a complementary In-context selection control. Both retain SIPA’s parallel actor and proposer, trial protocol, and budget. The filter takes the actor’s highest-ranked candidate whose action and arguments match a trial with no error or explicit failure. If none qualifies, it takes the original top choice. In-context selection gives the actor the raw trial outcomes and lets it reconsider its original candidates.

Method · accuracy (%)ToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only15.9816.0219.1776.1257.8937.04
Reflexion15.5320.3125.8372.3931.5833.13
SMURFS14.162.342.9227.6152.6319.93
Critic-only5.483.916.6720.9081.5823.71
In-context selection16.8918.7525.8382.8478.9544.65
Failure filtering17.3517.1923.7579.8568.4241.31
SIPA18.7220.3130.0087.3186.8448.64

Both controls improved on Actor-only. SIPA exceeded both on every benchmark, averaging 7.33 percentage points above Failure filtering and 3.99 above In-context selection. Two tool calls can both succeed while only one retrieves the requested entity. The filter then preserves the original ranking, whereas APA can use the returned content to downweight the less relevant action. Even with the raw outcomes in context, the actor kept its original choice in 54.7% of the steps where that choice was wrong. Similar persistence after disconfirming evidence has been observed in belief revision [5]. Together, these comparisons show why the selection rule matters even with matched trial access. These results are reported in the revised manuscript (Table 1, p. 6, and Appendix C.1, Table 6, p. 18).

AuthorsOpenReview ↗

Response to Reviewer hR3j (Part 2)

W3a / Q2. Variation across four runs

Table 1 (p. 6) and Table 4 (p. 8) reported one run per configuration. Appendix C.3 (Table 11, p. 23) now reports four Qwen3-4B runs for all six methods. The last column reports average accuracy weighted by the size of each benchmark.

Qwen3-4B accuracy (%, mean ± SD over 4 runs)ToolSandboxScienceWorldBFCLALFWorldBabyAISize-weighted average
Actor-only16.21 ± 0.9516.31 ± 0.9819.69 ± 1.2475.56 ± 1.8759.21 ± 3.4027.99 ± 0.52
Critic-only5.71 ± 0.593.91 ± 0.846.88 ± 0.5420.71 ± 1.2779.61 ± 4.4910.94 ± 0.33
ReasoningBank18.26 ± 0.9912.70 ± 0.8120.31 ± 1.0481.53 ± 1.6565.13 ± 3.3128.78 ± 0.43
Reflexion15.30 ± 0.9519.53 ± 0.8425.42 ± 1.0872.39 ± 1.6133.55 ± 4.4928.66 ± 0.50
SMURFS13.93 ± 0.952.54 ± 0.503.13 ± 0.5427.05 ± 1.2751.32 ± 3.4011.30 ± 0.34
SIPA18.61 ± 0.7820.41 ± 0.9829.58 ± 0.9086.75 ± 1.2785.53 ± 3.4035.26 ± 0.44

SIPA achieved 35.26 ± 0.44% task-weighted accuracy, ahead of Actor-only by 7.27 percentage points and the strongest baseline, ReasoningBank, by 6.48. It also had the highest four-run mean on each benchmark.

W3b / Q5. Scaling results on ScienceWorld

We confirmed the Actor-only results in Table 4 (p. 8) on the same 256 ScienceWorld tasks. Our code review found no parser defect, and the cause of the decline remains unclear. To test whether SIPA’s gain depends on this weak baseline, we added ReasoningBank and Reflexion at both Qwen3 scales and evaluated the same five methods on a second model family. Appendix C.3 (Tables 12–14, p. 24) reports accuracy (%) and the mean across benchmarks.

Qwen3-8B

MethodToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only20.554.3027.5063.4381.5839.47
Critic-only7.312.349.5852.2478.9530.08
ReasoningBank21.4618.7527.9282.0978.9545.83
Reflexion19.1825.3930.0075.3757.8941.57
SIPA20.0933.5928.7588.06100.0054.10

Qwen3-14B

MethodToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only23.295.8632.0870.9055.2637.48
Critic-only12.7910.1617.0875.37100.0043.08
ReasoningBank25.5722.2732.0885.8276.3248.41
Reflexion23.2929.6934.1782.0963.1646.48
SIPA26.9439.4534.5892.5497.3758.18

NVIDIA Nemotron-3-Nano-4B-BF16

MethodToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only14.6114.4516.6744.0355.2629.00
Critic-only5.944.307.9220.1573.6822.40
ReasoningBank16.8913.2822.9250.0063.1633.25
Reflexion15.5317.5825.4239.5539.4727.51
SIPA17.8119.5332.0858.2186.8442.89

On ScienceWorld, Reflexion was the strongest external baseline at both Qwen3 scales and improved with scale. SIPA still led Reflexion by 8.20 percentage points at 8B and 9.76 at 14B, and by 1.95 points on Nemotron. The advantage therefore persists against a stronger baseline across both model families.

Q3. Hyperparameter selection on ToolSandbox

We apologize for the ambiguity about the tuning data. SIPA requires no weight training, and its inference-time parameters had no established defaults, so we used the ToolSandbox training split as a development set. We selected (η,κ,G)=(32,10,6)(\eta,\kappa,G)=(32,10,6) on that split. Figure 5 (p. 9) shows the effect of varying one parameter at a time around those values. Appendix C.1 (p. 19) now documents the selection and evaluation protocol. We froze the configuration before evaluating the 219 held-out tasks and used it on the other four benchmarks without further tuning.

AuthorsOpenReview ↗

Response to Reviewer hR3j (Part 3)

W4a / Q4. Committed steps and total environment calls

The 69.6% is the ratio of committed steps in Table 1 (p. 6), as clarified in Appendix C.4 (p. 26). Across five benchmarks, SIPA averaged 15.76 such steps on successful tasks and ReasoningBank averaged 22.65. On ToolSandbox we also counted executions outside the live trajectory, including discarded trials. Committed calls include live retries. Table 16 (Appendix C.4, p. 26) averages each call column over the tasks solved by that method.

MethodAccuracy (%)Committed calls/successful taskAdditional calls/successful taskTotal calls/successful task
Actor-only15.9824.91024.91
Critic-only5.4827.0067.5094.50
Reflexion15.5324.3850.0074.38
SMURFS14.1623.9761.1385.10
SIPA18.728.5120.4428.95

Including trials, SIPA used 28.95 calls per successful task, 4.04 more than Actor-only and fewer than Critic-only, Reflexion, or SMURFS. On ToolSandbox, ReasoningBank’s 34.63 committed calls alone (Table 1, p. 6) exceeded SIPA’s total. Shorter live trajectories require fewer rounds of speculation, and the proposer chooses how many candidates to try. Although G=6G=6 is the ceiling, SIPA averaged only 2.40 additional calls per committed step.

We also evaluated both methods on all 219 ToolSandbox tasks under identical caps on total calls, counting each speculative trial, including those executed in parallel (Appendix C.4, Table 17, p. 26).

Total calls allowed per taskActor-only accuracy (%)SIPA accuracy (%)SIPA − Actor-only (pp)
3013.2411.87−1.37
5014.6115.98+1.37
10015.9817.81+1.83

Actor-only led at 30 calls, while SIPA achieved higher accuracy at 50 and 100. For inexpensive calls such as local search, elapsed time also matters. SIPA overlaps trials with actor reasoning (Fig. 4a, p. 8). Across the five benchmarks it gained 11.60 percentage points over Actor-only in mean accuracy, with a 10.2% increase in single-GPU latency per decision (11.81 to 13.02 seconds), as reported in Table 1 (p. 6) and Table 3 (p. 7).

W4b / Q6. Where isolated trials are feasible

We agree that irreversible actions limit where SIPA can speculate. SIPA requires isolated trials from the same decision state. Each candidate must leave the live task and the other branches unaffected. This covers read-only search, saved games, database snapshots, and code testing in separate workspaces. ALFWorld and ScienceWorld reconstruct the state by replaying committed actions in isolated branches. We discard the branches and execute the chosen action on the live trajectory.

Managing exploratory side effects is a shared constraint across speculative execution, execution-based search, and interactive agent learning. Speculative Actions discusses sandboxing [1]. Web tree search restores browser states [6], SWE-Search explores alternative repository edits [7], and SMURFS backtracks over planning memory and tool choices [8]. Recreated websites provide resettable replicas for agent learning [9]. Backtracking over a plan, however, cannot undo a completed transaction.

For live orders or messages, speculation requires a faithful preview that leaves the external system untouched. Appendix E.4 (p. 35) identifies this limit. If no preview is available, the agent can use the Actor-only policy for that decision.

References

[1] Ye et al. Speculative Actions: A Lossless Framework for Faster AI Agents. ICLR, 2026.

[2] Yao et al. Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends. ICLR, 2026.

[3] Peters et al. Relative Entropy Policy Search. AAAI, 2010.

[4] Abdolmaleki et al. Maximum a Posteriori Policy Optimisation. ICLR, 2018.

[5] Sadhu et al. DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark. EMNLP, 2026.

[6] Koh et al. Tree Search for Language Model Agents. TMLR, 2025.

[7] Antoniades et al. SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement. ICLR, 2025.

[8] Chen et al. Smurfs: Multi-Agent System using Context-Efficient DFSDT for Tool Planning. NAACL, 2025.

[9] Chae et al. Safe and Scalable Web Agent Learning via Recreated Websites. ICML, 2026.

Reviewer_aVduOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Speculative Inference-Time Policy Adaptation (SIPA), a training-free framework for improving language agents through immediate environment feedback at test time. While the actor generates candidate actions, a parallel proposer–critic branch performs speculative executions in isolated copies of the current environment. The resulting feedback is incorporated into the actor's action distribution through Asymmetric Policy Adaptation (APA), which penalizes potentially undesirable actions using a KL-constrained policy update. The authors also provide theoretical analyses of update stability and sufficient conditions for expected-return improvement.

Experiments on five interactive benchmarks demonstrate that SIPA achieves an average accuracy of 48.64% with Qwen3-4B, outperforming the strongest external baseline by 8.82 percentage points. Additional experiments examine ablations, efficiency, and scaling across Qwen3 models from 4B to 14B.

Critical strengths
  1. The idea of using immediate execution feedback to adapt action selection before commitment is intuitive and potentially useful.

  2. The paper provides theoretical analysis and empirical evaluations across multiple interactive benchmarks, with generally promising results.

Critical weaknesses
  1. Strong environment assumptions limit practical applicability. SIPA requires the ability to clone the current environment state and execute multiple candidate actions without affecting the ongoing environment. This assumption is difficult to satisfy in many real-world agent scenarios involving external APIs, irreversible actions, or shared environments. The paper does not sufficiently discuss this limitation.

  2. Insufficient comparisons under matched environment access. Unlike the external baselines, SIPA benefits from speculative execution of multiple actions at the current state. The Critic-only baseline is essentially an internal component comparison rather than an independent method with equivalent environment access. Without comparisons against simple one-step lookahead or execution-augmented action selection, it is unclear whether the improvements come from APA or merely from the additional environment feedback.

  3. Incomplete evaluation of interaction costs. The efficiency analysis emphasizes low wall-clock latency overhead but excludes speculative executions from the reported committed-step metrics. Since each decision may require multiple additional environment calls, the claimed efficiency advantages remain unclear under matched total interaction budgets.

  4. Limited generalization and scaling evaluation. All experiments use Qwen3 models, with the largest being 14B. Moreover, the scaling experiments only compare SIPA against Actor-only and Critic-only, without external baselines. This limits the evidence for generalization across model families and stronger foundation models.

Questions for authors
  1. How can SIPA be applied to real-world environments where state cloning or isolated speculative execution is unavailable?

  2. How does SIPA compare with simple one-step lookahead or action reranking methods given exactly the same speculative execution feedback?

  3. Does SIPA maintain its performance advantage when all speculative environment calls are included in the interaction budget?

  4. Can the authors evaluate SIPA on stronger or different model families and include representative external baselines in the scaling experiments?

6 comments
AuthorsOpenReview ↗

Response to Reviewer aVdu (Part 1)

Thank you for your constructive comments on environment access, interaction costs, and evaluation across model families. We address each point below.

W1 / Q1. Where isolated trials are feasible

SIPA requires that candidate actions can be tried from the same decision state without leaving persistent side effects on the ongoing task. This covers read-only web and document search, games with save states, patch testing in isolated workspaces, and queries against database snapshots. Where native cloning is unavailable, a sandbox can reconstruct the decision state from a snapshot or by replaying the committed history. If a live action is irreversible and no sandbox or preview is available, the agent can skip speculation and select from the original actor policy for that decision.

Managing the effects of exploratory actions is a shared constraint across several families of agents. In speculative execution, Speculative Actions discusses sandboxing actions taken before commitment [1]. In execution-based search, web agents reset and replay browser actions to revisit a decision [2], SWE-Search explores competing repository edits [3], and SMURFS backtracks over planning memory and tool choices [4]. In interactive agent learning, recreated websites let agents explore in resettable replicas [5]. These examples illustrate why the ability to isolate the effects of a trial determines where execution-based exploration can be deployed.

Thank you for highlighting this limitation. The new Limitations and Future Work section (Appendix E.4, p. 35) states the isolation requirement and explains why it rules out irreversible actions whose effects cannot be faithfully previewed.

W2 / Q2. Action selection with matched trial access

We apologize for this misunderstanding. Several baselines also use execution feedback, as described in Appendix D (p. 28). Before committing to an action, Critic-only executes the proposer’s candidates in isolated branches from the current state, scores them using the observed outcomes, and samples from the critic’s distribution over the tested actions. Reflexion uses execution feedback from one attempt to guide the next, and SMURFS uses tool responses to verify a plan and backtrack. Across the five benchmarks, SIPA reaches 48.64% mean accuracy, compared with 23.71% for Critic-only, 33.13% for Reflexion, and 19.93% for SMURFS.

Following the reviewer's suggestion, we added two controls that retain SIPA’s parallel actor and proposer branches, trial protocol, and budget. In the requested one-step lookahead control, In-context selection, the actor reads the raw trial outcomes alongside its original reasoning and chooses from its original candidates. In Failure filtering, it selects its highest-ranked candidate whose matching trial returned neither an execution error nor an explicit failure. If none qualifies, it keeps its original top choice.

Method · accuracy (%)ToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only15.9816.0219.1776.1257.8937.04
Reflexion15.5320.3125.8372.3931.5833.13
SMURFS14.162.342.9227.6152.6319.93
Critic-only5.483.916.6720.9081.5823.71
In-context selection16.8918.7525.8382.8478.9544.65
Failure filtering17.3517.1923.7579.8568.4241.31
SIPA18.7220.3130.0087.3186.8448.64

SIPA outperformed both controls on every benchmark, by an average of 3.99 percentage points over In-context selection and 7.33 over Failure filtering. In In-context selection, the actor kept its original action in 54.7% of the decision steps where its initial choice was wrong, even after seeing the trial outcomes. Persistence after disconfirming evidence has also been observed in controlled studies of belief revision [6]. APA gives execution evidence a direct role in the choice. A rejection penalty lowers the odds of the initial favorite relative to less penalized candidates, while the actor’s original ranking still guides the choice among equally penalized alternatives. This lets a trial correct a poor first choice even when it gives little guidance about the best replacement. These results are reported in the revised manuscript (Table 1, p. 6, and Appendix C.1, Table 6, p. 18).

AuthorsOpenReview ↗

Response to Reviewer aVdu (Part 2)

W3 / Q3. Accuracy under total interaction budgets

Thank you for suggesting a comparison under matched interaction budgets. Figure 9 (p. 25) shows that SIPA outperformed Actor-only on ToolSandbox at limits of 30, 50, and 100 committed actions per task. To test whether this advantage persists when speculative trials also count toward the budget, we evaluated both methods on the same tasks with limits of 30, 50, and 100 total environment calls per task, including speculative trials executed in parallel (Appendix C.4, Table 17, p. 26).

Total calls allowed per taskActor-only accuracy (%)SIPA accuracy (%)SIPA − Actor-only (pp)
3013.2411.87−1.37
5014.6115.98+1.37
10015.9817.81+1.83

SIPA trailed Actor-only by 1.37 percentage points at 30 calls, then led by 1.37 points at 50 and 1.83 points at 100. At the two higher budgets, SIPA achieved higher accuracy than Actor-only even with all speculative trials included.

To explain this trade-off, we also counted calls for SIPA and four baselines under the original ToolSandbox protocol (Appendix C.4, Table 16, p. 26). A committed call executes an action on the live trajectory and corresponds to one Step in Table 1 (p. 6). Additional calls are executions outside that trajectory, including speculative trials and discarded attempts.

MethodToolSandbox accuracy (%)Committed calls/successful taskAdditional calls/successful taskTotal calls/successful task
Actor-only15.9824.91024.91
Critic-only5.4827.0067.5094.50
Reflexion15.5324.3850.0074.38
SMURFS14.1623.9761.1385.10
SIPA18.728.5120.4428.95

SIPA averaged 28.95 total calls per successful task, 4.04 more than Actor-only and well below the 74.38 used by Reflexion, the least costly of the three other methods that try alternative actions. Two features limit the overhead. First, shorter committed trajectories require fewer rounds of speculation. SIPA averaged 8.51 committed calls, saving 16.40 relative to Actor-only and offsetting most of its 20.44 additional calls. Second, G=6G=6 is a ceiling. The proposer chooses how many candidates to generate at each state, and SIPA averaged only 2.40 additional calls per committed step.

In applications such as local document search, read-only catalog queries, and lightweight game simulations, a trial can be cheap relative to model inference. SIPA runs trials while the actor deliberates, so much of their latency overlaps with the actor’s reasoning (Fig. 4a, p. 8). On one GPU, this raised mean latency per decision from 11.81 to 13.02 seconds, an increase of 1.21 seconds or 10.2% (Table 3, p. 7). Across the five benchmarks, mean accuracy rose from 37.04% to 48.64%, a gain of 11.60 percentage points (Table 1, p. 6).

AuthorsOpenReview ↗

Response to Reviewer aVdu (Part 3)

W4 / Q4. Scaling and generalization across model families

We added ReasoningBank and Reflexion to the Qwen3-8B and Qwen3-14B evaluations and tested the same five methods on a second model family, NVIDIA Nemotron-3-Nano-4B-BF16. ReasoningBank was the strongest external baseline on average in our original Qwen3-4B comparison. Reflexion provides a complementary comparison with a method that uses feedback across attempts. Appendix C.3 (Tables 12–14, p. 24) reports task accuracy (%) on each benchmark and the unweighted mean across all five.

Qwen3-8B

MethodToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only20.554.3027.5063.4381.5839.47
Critic-only7.312.349.5852.2478.9530.08
ReasoningBank21.4618.7527.9282.0978.9545.83
Reflexion19.1825.3930.0075.3757.8941.57
SIPA20.0933.5928.7588.06100.0054.10

Qwen3-14B

MethodToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only23.295.8632.0870.9055.2637.48
Critic-only12.7910.1617.0875.37100.0043.08
ReasoningBank25.5722.2732.0885.8276.3248.41
Reflexion23.2929.6934.1782.0963.1646.48
SIPA26.9439.4534.5892.5497.3758.18

ReasoningBank had the highest mean accuracy among the external baselines at both scales. SIPA exceeded it by 8.27 percentage points at 8B and 9.77 at 14B. On ALFWorld, for example, its lead over ReasoningBank was 5.97 points at 8B and 6.72 at 14B.

NVIDIA Nemotron-3-Nano-4B-BF16

MethodToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only14.6114.4516.6744.0355.2629.00
Critic-only5.944.307.9220.1573.6822.40
ReasoningBank16.8913.2822.9250.0063.1633.25
Reflexion15.5317.5825.4239.5539.4727.51
SIPA17.8119.5332.0858.2186.8442.89

On Nemotron, SIPA led on all five benchmarks. Its 42.89% mean accuracy exceeded Actor-only by 13.89 percentage points and ReasoningBank by 9.64. The gain therefore extends to a second model family at 4B and remains present against external baselines at both larger Qwen3 scales.

References

[1] Ye, N., Ahuja, A., Liargkovas, G., Lu, Y., Kaffes, K., & Peng, T. (2026). Speculative Actions: A Lossless Framework for Faster AI Agents. International Conference on Learning Representations.

[2] Koh, J. Y., McAleer, S., Fried, D., & Salakhutdinov, R. (2025). Tree Search for Language Model Agents. Transactions on Machine Learning Research.

[3] Antoniades, A., Örwall, A., Zhang, K., Xie, Y., Goyal, A., & Wang, W. (2025). SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement. International Conference on Learning Representations.

[4] Chen, J., Liang, J., & Wang, B. (2025). Smurfs: Multi-Agent System using Context-Efficient DFSDT for Tool Planning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3281–3298.

[5] Chae, H., Park, J., & Ritter, A. (2026). Safe and Scalable Web Agent Learning via Recreated Websites. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306, 12294–12331.

[6] Sadhu, J., Shahad, S., & Marino, K. (2026). DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pp. 36952–36985.

Reviewer_S8EzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SIPA is a training-free way to improve an LLM agent's action choice at test time. While the actor model reasons about its next action, a lighter proposer branch suggests candidate actions and actually executes each one in an isolated copy (snapshot) of the current env state. A critic looks at the outcomes and scores which actions to avoid. Then APA (asymmetric policy adaptation) applies a KL-regularized exponential tilt to the actor's distribution over its own candidates, which only ever pushes probability down for actions the critic rejects. All three roles share one frozen Qwen3 model. There's some theory (stability under feedback error, a sufficient condition for local return improvement) and experiments on ToolSandbox, ScienceWorld, BFCL, ALFWorld and BabyAI-Text, where SIPA gets 48.6% average vs 39.8% for the best baseline (ReasoningBank) at 4B, with gains holding at 8B and 14B.

Critical strengths
  1. The empirical gains over Actor-only are large on the interactive envs. ALFWorld, ScienceWorld and BabyAI improve at 4B, 8B and 14B, and the paired bootstrap CIs in Table 10 clearly exclude zero there. Running the trials in parallel with actor reasoning so latency only goes up about 10% is a nice systems touch.
  2. The asymmetric design, where the critic can only veto and never promotes, is a sensible way to use a weak critic, and the ablations (no negative probe, no exec evidence) do back it up.
Critical weaknesses
  1. The key assumption is that you can snapshot the env and try actions for free in isolated copies. That's a big deal, and the comparison with the baselines doesn't really account for it. With env cloning, SIPA is basically doing one-step lookahead with a real simulator, which ReasoningBank, ACE, Reflexion etc. don't get. In most real agent settings (real APIs, anything with side effects, web, a user in the loop) you can't fork the state. So I'm not convinced the headline "+22% over the strongest baseline" compares like with like. The natural baseline is a simple lookahead: execute the candidates, show the actor the trial observations, and let it pick (or just pick whichever candidate makes progress). The ablations get partway there, but none of them is a plain "actor + trial results in context" baseline.
  2. The statistical support is uneven, and the framing oversells it. Everything looks like one run per config. BabyAI has 38 tasks, so one task is 2.6 points, and that's where the biggest gain is. The update strength η, KL cap κ and candidate count G were picked by sweeping on ToolSandbox, which is also an eval benchmark. On BFCL and ToolSandbox at 8B/14B the gain over Actor-only isn't significant (e.g. +34/−31 tasks on BFCL at 8B). "Steps" only averages over successful tasks and doesn't count the speculative executions. Calling this "test-time RL" is also a stretch. Nothing is learned or carried over; it's reranking a candidate list using one-step rollouts. The theory is mostly standard KL-regularized exponential-tilt results, which is fine, but it isn't much of a contribution on its own.
Questions for authors
  • How is env snapshotting done for each benchmark (e.g. ToolSandbox, BFCL)? Is it exact? What would SIPA look like in an env where you can't clone state?
  • Can you add a baseline where the actor sees the same trial observations directly in its prompt and picks an action (no critic, no APA)? That would tell us whether APA itself matters or the extra information does.
  • How many executions does SIPA use in total (committed + speculative) compared to the baselines? Could you report accuracy against total env interactions?
  • Were hyperparameters tuned anywhere other than ToolSandbox? Do the results hold on a held-out split?
  • Why is Actor-only at 8B so low on ScienceWorld (4.3%) compared to 4B (16.0%)? Is that a formatting/parsing failure?
4 comments
AuthorsOpenReview ↗

Response to Reviewer S8Ez (Part 1)

Thank you for the constructive feedback on the comparisons and the scope of our claims. We address each point below.

W1 / Q2. Isolated trials and matched feedback

SIPA assumes that candidate actions can be tried from the same state without affecting the live task, which covers a range of applications including web and document search, database queries on snapshots, games with save states, and sandboxed code testing. We have clarified this scope in Appendix E.4 (p. 35). Managing the side effects of exploratory actions is a shared constraint across speculative execution [1], execution-based tree search [2], and agent learning on resettable website replicas [3].

We would like to clarify that several baselines also use execution feedback (Appendix D, p. 28). Reflexion uses feedback across attempts, SMURFS uses tool responses to backtrack, and Critic-only tries proposals at the current state before sampling from the critic’s scores. We therefore compared two action-selection rules under matched trial access. In In-context selection, the actor reads the raw outcomes and chooses from its original candidates. Failure filtering takes its highest-ranked candidate whose trial returned no error or explicit failure, falling back to its original top choice if none qualifies.

Method · accuracy (%)ToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only15.9816.0219.1776.1257.8937.04
Reflexion15.5320.3125.8372.3931.5833.13
SMURFS14.162.342.9227.6152.6319.93
Critic-only5.483.916.6720.9081.5823.71
In-context selection16.8918.7525.8382.8478.9544.65
Failure filtering17.3517.1923.7579.8568.4241.31
SIPA18.7220.3130.0087.3186.8448.64

Giving the actor the observations directly raised mean accuracy from 37.04% to 44.65%. With matched access, SIPA exceeded In-context selection by 3.99 percentage points and Failure filtering by 7.33, outperforming both on every benchmark. In In-context selection, the actor kept its original action in 54.7% of the decision steps where its initial choice was wrong, even after seeing the trial outcomes. Persistence after disconfirming evidence has also been observed in controlled studies of belief revision [4]. APA lets evidence against the actor’s favorite reduce its odds while preserving preferences among equally penalized alternatives. A trial can thus correct the first choice even when it leaves the best replacement unresolved. These results are reported in the revised manuscript (Table 1, p. 6, and Appendix C.1, Table 6, p. 18).

W2a. Accuracy across runs and benchmarks

We repeated the Qwen3-4B evaluation four times. In the last column each task (not benchmark) carries equal weight, so BabyAI accounts for only 38 of 887 tasks (4.3%). Appendix C.3 (Table 11, p. 23) reports mean accuracy and its standard deviation across runs.

Qwen3-4B accuracy (%, mean ± SD over 4 runs)ToolSandboxScienceWorldBFCLALFWorldBabyAISize-weighted average (887 tasks)
Actor-only16.21 ± 0.9516.31 ± 0.9819.69 ± 1.2475.56 ± 1.8759.21 ± 3.4027.99 ± 0.52
Critic-only5.71 ± 0.593.91 ± 0.846.88 ± 0.5420.71 ± 1.2779.61 ± 4.4910.94 ± 0.33
ReasoningBank18.26 ± 0.9912.70 ± 0.8120.31 ± 1.0481.53 ± 1.6565.13 ± 3.3128.78 ± 0.43
Reflexion15.30 ± 0.9519.53 ± 0.8425.42 ± 1.0872.39 ± 1.6133.55 ± 4.4928.66 ± 0.50
SMURFS13.93 ± 0.952.54 ± 0.503.13 ± 0.5427.05 ± 1.2751.32 ± 3.4011.30 ± 0.34
SIPA18.61 ± 0.7820.41 ± 0.9829.58 ± 0.9086.75 ± 1.2785.53 ± 3.4035.26 ± 0.44

Under this weighting, SIPA led Actor-only by 7.27 percentage points and ReasoningBank, the strongest baseline, by 6.48. SIPA also had the highest four-run mean on each benchmark.

References

[1] Ye, N., Ahuja, A., Liargkovas, G., Lu, Y., Kaffes, K., & Peng, T. (2026). Speculative Actions: A Lossless Framework for Faster AI Agents. International Conference on Learning Representations.

[2] Koh, J. Y., McAleer, S., Fried, D., & Salakhutdinov, R. (2025). Tree Search for Language Model Agents. Transactions on Machine Learning Research.

[3] Chae, H., Park, J., & Ritter, A. (2026). Safe and Scalable Web Agent Learning via Recreated Websites. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306, 12294–12331.

[4] Sadhu, J., Shahad, S., & Marino, K. (2026). DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pp. 36952–36985.

AuthorsOpenReview ↗

Response to Reviewer S8Ez (Part 2)

W2b / Q4. Hyperparameter selection and transfer

We apologize for the ambiguity about the tuning data. SIPA requires no weight training, and its inference-time parameters had no established defaults. We therefore used the ToolSandbox training split as a development set to choose (η,κ,G)=(32,10,6)(\eta,\kappa,G)=(32,10,6). Figure 5 (p. 9) shows sensitivity around these values. Appendix C.1 (p. 19) now specifies how the configuration was selected and transferred.

We froze the configuration before evaluating the 219 held-out ToolSandbox tasks and used the same configuration on the other four benchmarks without further hyperparameter tuning for the main evaluation.

W2c / Q3. Total environment calls and matched budgets

We counted calls for SIPA and four baselines on ToolSandbox (Appendix C.4, Table 16, p. 26). A committed call executes an action on the live trajectory and corresponds to one Step in Table 1 (p. 6). Additional calls are executions outside that trajectory, including speculative trials and discarded attempts.

MethodToolSandbox accuracy (%)Committed calls/successful taskAdditional calls/successful taskTotal calls/successful task
Actor-only15.9824.91024.91
Critic-only5.4827.0067.5094.50
Reflexion15.5324.3850.0074.38
SMURFS14.1623.9761.1385.10
SIPA18.728.5120.4428.95

SIPA averaged 28.95 total calls per successful task, 4.04 more than Actor-only and fewer than any of the three other methods that try alternatives. Two factors limit the overhead. First, shorter trajectories require fewer rounds of speculation. SIPA used 8.51 committed calls, saving 16.40 relative to Actor-only and offsetting most of its 20.44 additional calls. Second, G=6G=6 is a ceiling. The proposer chooses how many candidates to generate at each state, and SIPA averaged 2.40 additional calls per committed step.

To measure accuracy under matched budgets, we evaluated both methods on all 219 ToolSandbox tasks with identical caps on total calls per task. These totals include speculative trials executed in parallel (Appendix C.4, Table 17, p. 26).

Total calls allowed per taskActor-only accuracy (%)SIPA accuracy (%)SIPA − Actor-only (pp)
3013.2411.87−1.37
5014.6115.98+1.37
10015.9817.81+1.83

At 30 calls per task, Actor-only led by 1.37 percentage points. At 50 and 100 calls, SIPA led by 1.37 and 1.83 points. SIPA thus retained its advantage at both higher budgets with all trials counted.

When a trial is cheap relative to model inference, as in local document search, catalog queries, or lightweight game simulations, users may care more about accuracy and response time than about the number of calls. Trial execution overlaps with actor reasoning (Fig. 4a, p. 8). On the same GPU, mean latency per decision increased from 11.81 to 13.02 seconds, an overhead of 1.21 seconds or 10.2% (Table 3, p. 7). Across the five benchmarks, mean task accuracy rose from 37.04% to 48.64%, a gain of 11.60 percentage points (Table 1, p. 6).

W2d / W2e. Test-time policy improvement

At each decision, SIPA optimizes the probability distribution over the actor’s candidate actions. Trial outcomes supply a proxy advantage, and APA chooses the distribution that maximizes expected proxy advantage with KL regularization toward the actor prior (Proposition 1, p. 4, and Appendix B.1, p. 15). The agent acts under that updated policy immediately, with model weights frozen. JitRL [5] similarly interprets a change in action logits as policy improvement, using advantages estimated from past returns. Adaptation can also be temporary in weight-based test-time RL, as when pretrained weights are restored after a deployment interval [6].

Section 5 (p. 9) places APA in the context of regularized policy optimization, which also has established connections to search [7]. The theoretical question specific to SIPA is when evidence about an immediate outcome can improve a decision whose consequences unfold over many steps. Theorem 2 (p. 5) bounds how far errors in the correction signal can move the policy from the update under reference feedback. Theorem 3 (p. 5) and Eq. (19) in Appendix B.3 (p. 16) relate that change in action probabilities to continuation values under the original actor. Rejecting a candidate can improve expected return even when the critic cannot rank the remaining actions, provided the probability it redistributes goes to better continuations on average. Corollary 4 (Appendix B.3, p. 17) connects this criterion to task performance by expressing the expected return difference as the sum of local gains in expectation along the adapted policy’s trajectories. This identifies both the value of selective rejection and its limit when immediate feedback is misleading about downstream consequences.

AuthorsOpenReview ↗

Response to Reviewer S8Ez (Part 3)

Q1. State reconstruction and isolated execution

Each branch reconstructs the current decision state and tries one candidate. CPU workers execute branches while the actor deliberates. After collecting feedback, we discard the branches and execute the selected action on the live trajectory (Appendix D, Algorithm 1, p. 27). The table below gives the implementation details for each benchmark.

BenchmarkHow the branch reconstructs the decision state
ToolSandboxExport the currentExecutionContext as a dictionary, rebuild it withExecutionContext.from_dict(), and run the candidate tool call inside an independentnew_context().
BFCLDeserialize the current tool instances withpickle.loads() and invoke the official executor on the restored objects.
BabyAI-TextDeserialize the environment and its wrappers, including the mission verifier and RNG state.
ALFWorldReopen the same TextWorld game file and replay the committed commands.
ScienceWorldStart a separate Java environment, load the same task and variation, and replay the interaction history.

We compare restored ToolSandbox contexts against their serialized state and check ALFWorld fingerprints that include physical facts. During ScienceWorld replay we compare observations, termination status, and available-action summaries. These checks establish fidelity on the inspected fields.

Appendix E.4 (p. 35) specifies the requirement for isolated trial execution from the same decision state. A saved game, database snapshot, or replay in a sandbox can supply this capability without a native clone operation. If an action would send a real message or place a live order and no isolated preview exists, the agent can skip speculation and sample from its original actor distribution.

Q5. Actor-only accuracy on ScienceWorld

The drop in ScienceWorld accuracy at 8B warranted a closer look. We checked the Actor-only implementation and confirmed the accuracies in Table 4 (p. 8) on the same 256 tasks. Our code review did not identify a parser defect.

To test whether SIPA’s gains extend to larger models and another model family, we added ReasoningBank and Reflexion to the Qwen3-8B and Qwen3-14B comparisons and evaluated all five methods on NVIDIA Nemotron-3-Nano-4B-BF16. Appendix C.3 (Tables 12–14, p. 24) reports accuracy (%) on each benchmark and the mean across all five.

Qwen3-8B

MethodToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only20.554.3027.5063.4381.5839.47
Critic-only7.312.349.5852.2478.9530.08
ReasoningBank21.4618.7527.9282.0978.9545.83
Reflexion19.1825.3930.0075.3757.8941.57
SIPA20.0933.5928.7588.06100.0054.10

Qwen3-14B

MethodToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only23.295.8632.0870.9055.2637.48
Critic-only12.7910.1617.0875.37100.0043.08
ReasoningBank25.5722.2732.0885.8276.3248.41
Reflexion23.2929.6934.1782.0963.1646.48
SIPA26.9439.4534.5892.5497.3758.18

NVIDIA Nemotron-3-Nano-4B-BF16

MethodToolSandboxScienceWorldBFCLALFWorldBabyAIMean
Actor-only14.6114.4516.6744.0355.2629.00
Critic-only5.944.307.9220.1573.6822.40
ReasoningBank16.8913.2822.9250.0063.1633.25
Reflexion15.5317.5825.4239.5539.4727.51
SIPA17.8119.5332.0858.2186.8442.89

On ScienceWorld, SIPA exceeded Reflexion, the strongest external baseline at both Qwen3 scales, by 8.20 percentage points at 8B and 9.76 at 14B. On Nemotron, SIPA led on all five benchmarks and achieved 42.89% mean accuracy, 9.64 points above ReasoningBank.

References (continued)

[5] Li, Y., Lin, Z., Deng, A., Zhang, X., He, Y., Ji, S., Cao, T., & Hooi, B. (2026). Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306, 68204–68230.

[6] Bagatella, M., Albaba, M., Hübotter, J., Martius, G., & Krause, A. (2026). Test-time Offline Reinforcement Learning on Goal-related Experience. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306, 5200–5218.

[7] Grill, J.-B., Altché, F., Tang, Y., Hubert, T., Valko, M., Antonoglou, I., & Munos, R. (2020). Monte-Carlo Tree Search as Regularized Policy Optimization. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, 119, 3769–3778.

Thank you again for the constructive comments. Please let us know if our responses or the revised manuscript leave any questions unresolved. We would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.