acceptodds
Under review as a conference paper at ICLR 2027

vSFT: Learning to Verbally Tune Agents at Test Time

Abstract

Large language model agents are increasingly improved via agentic reinforcement learning, which can be on-policy but suffers from expensive rollouts and sparse rewards, limiting its application at test time. In this paper, we present a verbally Supervised Fine-Tuning (vSFT) framework that enables the conjunction of dense feedback in natural language and weight updates at test time. Specifically, our vSFT forms a closed loop between an agent and a supervisor via a hypernetwork, in which the supervisor generates verbal supervision from the agent's failed trajectories, and the hypernetwork converts that supervision into low-rank weight updates to tune the agent. We validate that this test-time in-parameter evolving process outperforms both its in-context counterparts and prevalent reinforcement learning methods. In particular, our vSFT using Qwen3-4B outperforms GRPO and Reflexion by 11.5% and 10.8% in success rate on ALFWorld's in-distribution split, and by 4.8% and 18.2% on WebShop, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM Agents, Self-Improvement, Hypernetworks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_ffTtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a verbally supervised fine-tuning framework that enables dense natural-language feedback and test-time weight updates. The framework supports on-policy, in-parameter learning from dense feedback without backpropagation, through a closed-loop interaction between an agent and a supervisor. Experiments on ALFWorld and WebShop demontrate that Qwen3-4B with the proposed framework can outperform several baseline methods.

Critical strengths

The framework enables on-policy, in-parameter learning with dense natural-language feedback without requiring backpropagation. The core idea is novel, particularly the closed-loop interaction between the agent and supervisor for test-time adaptation.

Critical weaknesses
  1. Several training and evaluation details remain unclear. For example, the reported time cost is lower than that of competing methods, but differences in the number of trials and the supervisor's access to agent trajectories may affect the fairness of the comparisons. These factors require further clarification.

  2. The experiments are limited to a single model family. Evaluation across different model families would help establish the generalizability of the proposed framework.

Questions for authors

How are the interaction budgets and supervisor access to agent trajectories controlled to ensure fair comparisons with baselines? Does the proposed framework generalize to other model families?

5 comments
AuthorsOpenReview ↗

Response to Weakness 1

W1: Several training and evaluation details remain unclear. Differences in the number of trials and the supervisor's access to agent trajectories may affect the fairness of the comparisons.

Thank you for your careful reading and constructive suggestions.

To address this concern, we clarify the experimental settings and conduct additional experiments to ensure fairer comparisons.


Inference budgets

All methods, including RL baselines, use the same environment interaction limits per trial: 50 steps on ALFWorld and 15 steps on WebShop. However, as the reviewer noted, RL methods use only one trial per task, whereas vSFT and Reflexion allow up to 10 trials. This presents a trade-off between training and inference costs. RL incurs substantial training costs but typically requires only one inference attempt, whereas Reflexion requires no additional training but relies on repeated test-time reflection. vSFT lies between them in training costs (higher than Reflexion but substantially lower than RL), while introducing a more complex inference procedure. We provide a wall-clock time comparison below:

  • Training overhead. On ALFWorld, using a single RTX 6000D (84 GB) GPU, GRPO requires 62.3 hours per epoch over 10 epochs. In contrast, vSFT requires 9.1 hours per epoch over 5 epochs, plus 8.3 hours for data collection. This is a 91.4% reduction in total training time.
  • Inference overhead. On ALFWorld's seen split, using a single RTX 4090 (48 GB) GPU, Reflexion takes 591.8s, while vSFT takes 607.7s (2.7% longer). LoRA generation and loading account for 60.9s (10.0%) and 4.1s (0.7%), respectively.

Regarding vSFT's substantial training savings, we provide a detailed analysis in our response to Reviewer rapH, and a breakdown of its inference costs in our response to Reviewer kbbX. We kindly refer the reviewer to that response and would be happy to elaborate further if needed.


Fairer comparisons

In the original paper, we have made efforts to ensure fair comparisons. For example, we introduce skills to make the in-context baseline (i.e., Reflexion) as strong as possible. For the same reason, we use a carefully designed, instructive system prompt (Appendix C). These designs help isolate the gains attributable to vSFT.

Nevertheless, we thank the reviewer for identifying two remaining fairness concerns:

  1. mismatched test-time attempt budgets between vSFT and RL methods;
  2. the absence of reflections for RL methods, where a supervisor has access to agent trajectories and provides feedback.

We therefore compare vSFT against GRPO and GiGPO on ALFWorld under matched test-time attempts in Table R1. We also allow the GiGPO model to reflect on its trajectories, in a Reflexion/vSFT way. These experiments reveal two advantages of vSFT over RL:

  1. vSFT outperforms GRPO/GiGPO under matched attempts. With fewer test-time attempts per task, vSFT still substantially outperforms both GRPO and GiGPO. Given the observed performance saturation of RL models, we do not allocate more attempts.

  2. vSFT is complementary to RL. Although we compare vSFT with RL, it is not limited to base models and can also be applied to RL models. Notably, our hypernetwork trained on the base model can still effectively improve its GiGPO model, yielding greater gains than in-context reflection (i.e., Reflexion). Thus, GRPO and GiGPO are complementary to, rather than direct alternatives to, vSFT; Reflexion is a more direct baseline.

Table R1. Comparisons under matched test-time attempts on ALFWorld. We evaluate GRPO and GiGPO with three attempts at each of four temperatures (0, 0.1, 0.2, and 0.3). Methods with reflection are allowed 10 additional reflection trials.

MethodSeenUnseenAvg.
GRPO85.086.685.8
GiGPO87.984.386.1
GiGPO+Reflexion87.985.186.5
GiGPO+vSFT90.087.388.7
GiGPO+vSFT$^\dagger$90.790.390.5
GiGPO+vSFT$^\ddagger$90.791.090.9
vSFT92.992.592.7
vSFT$^\dagger$96.497.096.7
vSFT$^\ddagger$99.3100.099.7
AuthorsOpenReview ↗

Response to Weakness 2

W2: The experiments are limited to a single model family.

Thank you for raising this important concern.

We acknowledge that empirical evaluations or theoretical analyses across model families would further support our method. Due to the limited time during rebuttal, it is hard to conduct additional experiments with a different model family.

Nevertheless, we would like to provide a theoretical perspective on the underlying mechanism of vSFT, offering insights into why it is expected to work across different models. Moreover, we conduct out-of-domain experiments to provide evidence that vSFT learns a generalizable behavior-steering mechanism.

Neural thickets and hypernetwork. Recent work [6] suggests that task-specific experts may be densely distributed in the parameter space surrounding well-pretrained models (validated across model families, including Qwen, Llama, and OLMo3), a phenomenon termed neural thickets. This finding motivates using random weight perturbations to discover such experts. From this perspective, vSFT can be viewed as a learnable, multi-round search mechanism in parameter space. Rather than relying on random perturbations, vSFT trains a hypernetwork to iteratively generate low-rank weight updates guided by verbal feedback, thereby steering the agent toward task-specific solutions. This interpretation offers a potential explanation for why the underlying mechanism could generalize across different model families.

Generalization to new domains. To examine generalization beyond ALFWorld/WebShop, we conduct additional evaluations on JSONSchemaBench [2] and SCBench [5]. As shown in Tables R2 and R6, vSFT consistently improves JSON/schema compliance, while KV caches computed with the adapter generated by vSFT for ALFWorld even achieve a higher average score on out-of-domain tasks. These results support both vSFT's transferable behavior steering ability and the cross-domain generalization of its adapted KV representations.

Table R2. JSONSchemaBench results. Each cell reports JSON compliance/schema compliance (%). The hypernetwork is trained on ALFWorld. We internalize the system prompt and few-shot examples, while keeping the specific schema in context.

MethodGitHub EasyGitHub MediumGitHub Hard
In-context98.3/88.290.4/66.967.1/32.3
D2L98.1/85.490.6/63.164.4/30.7
vSFT (ours)99.1/91.798.1/71.086.1/37.2

Table R6: SCBench results. Base KV denotes caches computed by the base model; LoRA KV denotes caches computed with the adapter generated by vSFT for ALFWorld. The base model performs decoding.

TaskBase KVLoRA KV
Retr.KV54.662.2
Retr.MultiHop40.032.0
Code.RepoQA53.653.0
En.QA27.324.5
Zh.QA3.44.0
En.MultiChoice17.829.7
Math.Find28.314.2
ICL.ManyShot70.770.7
En.Sum36.136.5
Mix.Sum+NIAH (Sum)36.437.3
Mix.Sum+NIAH (NIAH)65.083.6
Mix.RepoQA+KV (RepoQA)60.254.6
Mix.RepoQA+KV (KV)81.891.2
Average44.345.6 (+1.4)

[2] JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

[5] SCBench: A KV Cache-Centric Analysis of Long-Context Methods

[6] Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

AuthorsOpenReview ↗

Opportunity for Further Discussion

Dear Reviewer ffTt,

Thank you again for your constructive review. We hope our rebuttal, particularly the additional experiments, has addressed your concerns.

As the discussion period draws to a close, we would be grateful to know whether any concerns remain and would be happy to provide further clarification.

Thank you for your time and consideration.

Sincerely,
The Authors

Reviewer_kbbXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes vSFT, a test-time adaptation framework for large language model (LLM) agents. After an agent fails a task, a supervisor generates natural language feedback based on the failure trajectory. A pre-trained hypernetwork then uses the system prompt, few-shot examples, and accumulated feedback to generate a LoRA adapter for the agent. The hypernetwork is trained offline on condition-query-response data collected from successful trajectories. The authors report results on ALFWorld and WebShop, comparing vSFT with Reflexion and several reinforcement learning baselines. They also test different supervisor configurations and merge task-specific adapters to study transfer across data splits.

Critical strengths
  1. The paper presents a clear way to turn language feedback into parameter updates. Section 3.1 describes the loop between the agent, supervisor, and hypernetwork. Equation 10 gives the hypernetwork’s training objective. This provides a concrete alternative to keeping reflections only in the agent’s context.
  2. The paper evaluates vSFT on ALFWorld and WebShop, with several supervisor settings. Tables 2–4 and Figures 4–6 report results across trials and compare different supervisors. These experiments provide useful initial evidence about the method’s performance on the evaluated benchmarks.
  3. The one-trial and multi-trial vSFT results help separate the effect of generating an initial adapter from the gains made over later feedback rounds. Tables 2 and 3 report one-trial results, while Figures 4 and 5 show performance across trials. Note that one-trial vSFT already uses an adapter generated from the system prompt and few-shot examples.
  4. Figure 6 studies transfer between data splits of the same benchmark by merging task-specific adapters. The comparison with one-trial and multi-trial vSFT provides initial evidence about whether feedback encoded in the adapters can help on a held-out split.
Critical weaknesses
  1. The source and availability of the reference successful trajectories at evaluation time are unclear. Algorithm 1 takes a reference task and its successful trajectory as inputs, and Equation 3 shows that the supervisor uses them when generating feedback. The paper does not explain how these references are selected for evaluation, whether they overlap with the training data, or whether they are available for every test task. If the method depends on successful trajectories from similar tasks, this could limit its use in settings where such examples are unavailable.
  2. The paper should define its claims about test-time adaptation and on-policy learning more precisely. The hypernetwork is trained offline on data from successful trajectories. At test time, it generates LoRA adapters from the current condition and accumulated feedback, while the base agent and hypernetwork remain fixed. The results include ALFWorld’s unseen split and held-out splits in Figure 6, but they do not establish generalization to new domains or substantially different task distributions. The paper should distinguish this form of test-time adapter generation from online updates to the base model or hypernetwork, and clarify what it means by on-policy and self-improvement.
  3. The cost analysis does not show the total cost of vSFT. Appendix A compares the number of rollouts used for data curation with the rollout budget of a particular GRPO setup. It does not include hypernetwork training, supervisor API calls and token use, test-time trial costs, or the time needed to generate adapters. The reported numbers support a claim of fewer data-curation rollouts, but not a claim that vSFT has lower overall computational or economic cost. A comparison should separate one-time training costs from costs per test task.
  4. The paper does not report enough information to judge uncertainty in the results. Tables 2–4 and Figures 4–8 mainly show success rates or average scores without confidence intervals or variance estimates. The paper also does not clearly report random seeds and decoding settings for the main results. This makes it hard to assess the reliability of results on smaller evaluation sets or when methods have similar scores. The authors should report the evaluation sample size for each result and provide uncertainty estimates for the main comparisons.
Questions for authors
  1. How are the reference task and successful trajectory in Algorithm 1 selected during evaluation? Are they drawn from the training data, generated by an auxiliary model, or obtained another way? Do they overlap with the evaluation tasks? Please also report how performance changes when the reference trajectory is unavailable or is replaced with a trajectory from a different task.
  2. Please provide a cost breakdown for vSFT, including hypernetwork training, supervisor API calls and token use, test-time trials, and adapter generation overhead. Please distinguish one-time costs from per-task costs and compare them with GRPO using a consistent measure.
  3. Were the main results run with multiple random seeds? Please report the decoding settings, sample sizes, and confidence intervals or other uncertainty estimates for the main comparisons. If results vary with decoding temperature, please report that separately.
  4. Please provide more details about the hypernetwork training data and procedure, including the dataset composition and size, number of training epochs, stopping criteria, and other implementation settings needed to reproduce the results.
7 comments
AuthorsOpenReview ↗

Response to Weakness 1 & Question 1

W1 & Q1: vSFT's dependence on reference tasks and their successful trajectories, which may be difficult to obtain at test time, is unclear.

Thank you for your careful reading and insightful question.

We agree that the contribution of reference tasks and their successful trajectories was not sufficiently discussed or evaluated in the original paper. In this response, we address this concern from the following aspects:

  • Why we introduce refernece tasks. Note that both vSFT and Reflexion use reference tasks in our experiments. This design aims to make the in-context baseline (i.e., Reflexion) as strong as possible, rather than to enhance vSFT. For the same reason, we compile a carefully designed, instructive system prompt (Appendix C). This helps isolate vSFT's solid improvements.

  • How reference tasks are selected. During evaluation, we maintain a pool of successfully completed tasks, initially empty and therefore not overlapping with the training data. When generating skills, we randomly select a reference task from this pool and provide it to the supervisor.

  • How reference tasks are used. They are used only to generate skills expected to transfer across tasks, and they are not provided to the supervisor when generating reflections. Fig.2 illustrates that vSFT can also internalize raw experience (e.g., retrieved similiar tasks and their successful trajectories), but this is not used in our current implementation. Instead, our work focuses on test-time reflection internalization, and our response to Reviewer rapH (Table R3) shows that task-specific reflections are the primary driver of performance gains.

  • Performance without reference tasks. As shown in Table R5, removing reference tasks (effectively, skills generated from them) results in only minor performance drops, and vSFT shows even smaller decreases than Reflexion.

Table R5. Performance without skill internalization.

MethodALFWorld SeenALFWorld UnseenWebShopAvg. Change
Reflexion82.175.430.8N/A
Reflexion w/o skill80.068.731.0-2.9
Reflexion$^\dagger$89.377.644.8N/A
Reflexion$^\dagger$ w/o skill87.978.443.4-0.7
vSFT92.992.549.0N/A
vSFT w/o skill92.193.347.4-0.5
vSFT$^\dagger$96.497.059.8N/A
vSFT$^\dagger$ w/o skill95.096.360.8-0.4
vSFT$^\ddagger$99.3100.062.0N/A
vSFT$^\ddagger$ w/o skill98.6100.061.8-0.3
AuthorsOpenReview ↗

Response to Weakness 2

W2: The paper should distinguish this form of test-time adapter generation from online updates to the base model or hypernetwork, and clarify what it means by on-policy and self-improvement.

Thank you for your constructive suggestions!

To address this concern, we clarify below what we mean by on-policy adaptation and self-improvement. Meanwhile, we provide additional experimental evidence of vSFT's generalization to new domains.


On-policy adaptation

As the reviewer notes, the hypernetwork is trained offline and remains fixed at test time. We therefore do not claim on-policy adaptation of the hypernetwork itself. Instead, vSFT enables on-policy adaptation of the agent through an iterative feedback loop: the policy adapted by the hypernetwork in each trial generates a rollout, whose feedback guides the weight update in the next trial. Thus, although the base weights and hypernetwork remain fixed, the agent's effective weights evolve across trials. Broader terms such as test-time adaptation or test-time adapter generation do not explicitly capture this cross-trial dependence between successive weight updates.

An alternative on-policy implementation would retrain the hypernetwork for the adapted model at each trial, such that each LoRA is applied to the adapted model rather than the base model. However, this would introduce repeated training overhead. Instead, vSFT keeps the hypernetwork fixed and accumulates feedback across trials, from which it generates adapters relative to the base model. The following equation illustrates the conceptual analogy of vSFT to this implementation (please refresh if the equation is not displayed):

θ→hϕ(z(0))θ(1)→hϕ(1)(z(1))θ(2)→hϕ(2)(z(2))θ(3)⏟On-policy adaptation by updating the hypernetwork hϕ≈θ→hϕ(z(0),z(1),z(2))θ(3)⏟On-policy adaptation by accumulating feedback z(0),z(1),z(2) (vSFT)\underbrace{\theta \xrightarrow{h_{\phi}(z^{(0)})} \theta^{(1)} \xrightarrow{h_{\phi^{(1)}}(z^{(1)})} \theta^{(2)} \xrightarrow{h_{\phi^{(2)}}(z^{(2)})} \theta^{(3)}}_{\text{On-policy adaptation by updating the hypernetwork }h_{\phi}} \quad\approx\quad \underbrace{\theta \xrightarrow{h_{\phi}(z^{(0)},z^{(1)},z^{(2)})} \theta^{(3)}}_{\text{On-policy adaptation by accumulating feedback }\\{z^{(0)},z^{(1)},z^{(2)}\\} \text{ (vSFT)}}

As discussed above, we use “on-policy” to indicate that each new feedback z(t)z^{(t)} is derived from a rollout of the current policy parameterized by θ(t)\theta^{(t)}. Similar to the recent finding [3,4] that a consistent update direction emerges from noisy RLVR rewards through mutual denoising, we expect our hypernetwork to similarly denoise accumulated feedback to directly generate the LoRA update from the base model to the final adapted model.

[3] On Predictability of Reinforcement Learning Dynamics for Large Language Models

[4] You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories


Self-improvement

As noted in the footnote on page 6 of our original paper, we define self-improvement as the ability of a system (e.g., the agent and hypernetwork together in our framework) to improve its own performance (i.e., "doing") in response to self-reflection or external feedback, rather than merely generate high-quality reflections (i.e., "knowing").


Generalization to new domains

To examine generalization beyond ALFWorld/WebShop, we conduct additional evaluations on JSONSchemaBench [2] and SCBench [5].

As shown in Tables R2 and R6, vSFT consistently improves JSON/schema compliance, while KV caches computed with the adapter generated by vSFT for ALFWorld even achieve a higher average score on out-of-domain tasks. These results support both vSFT's transferable behavior steering ability and the cross-domain generalization of its adapted KV representations.

Table R2. JSONSchemaBench results. Each cell reports JSON compliance/schema compliance (%). The hypernetwork is trained on ALFWorld. We internalize the system prompt and few-shot examples, while keeping the specific schema in context.

MethodGitHub EasyGitHub MediumGitHub Hard
In-context98.3/88.290.4/66.967.1/32.3
D2L98.1/85.490.6/63.164.4/30.7
vSFT (ours)99.1/91.798.1/71.086.1/37.2

Table R6: SCBench results. Base KV denotes caches computed by the base model; LoRA KV denotes caches computed with the adapter generated by vSFT for ALFWorld. The base model performs decoding.

TaskBase KVLoRA KV
Retr.KV54.662.2
Retr.MultiHop40.032.0
Code.RepoQA53.653.0
En.QA27.324.5
Zh.QA3.44.0
En.MultiChoice17.829.7
Math.Find28.314.2
ICL.ManyShot70.770.7
En.Sum36.136.5
Mix.Sum+NIAH (Sum)36.437.3
Mix.Sum+NIAH (NIAH)65.083.6
Mix.RepoQA+KV (RepoQA)60.254.6
Mix.RepoQA+KV (KV)81.891.2
Average44.345.6 (+1.4)

[2] JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

[5] SCBench: A KV Cache-Centric Analysis of Long-Context Methods

AuthorsOpenReview ↗

Response to Weakness 3 & Question 2

W3 & Q2: The paper lacks a detailed cost analysis of vSFT and comparisons with GRPO using a consistent measure.

To address this concern, we first provide a more detailed analysis of vSFT's training and inference costs, beyond the rollout counts reported in the original paper, showing that vSFT is substantially more efficient to train while maintaining comparable inference costs to its in-context counterpart. We then compare vSFT against GRPO and GiGPO on ALFWorld under matched test-time attempt budgets, showing that vSFT still substantially outperforms these RL methods and can also be applied to further improve them.


Detailed training and inference costs

  1. Training cost. Compared with GRPO, vSFT has a simpler training pipeline and requires substantially less training time for two reasons:

    • Decoupled data collection. Unlike on-policy RL methods such as GRPO, which interleave rollout collection with training updates, vSFT allows offline and off-policy data collection.
    • Simpler supervised optimization. Our hypernetwork is trained with standard cross-entropy loss on previously collected successful trajectories. In contrast, GRPO estimates relative advantages from rewards across multiple rollouts and optimizes a clipped policy objective. Thus, vSFT avoids online reward evaluation, advantage estimation, and policy-ratio clipping during training.

    On ALFWorld, using a single RTX 6000D (84 GB) GPU, GRPO requires 62.3 hours per epoch over 10 epochs. In contrast, vSFT requires 9.1 hours per epoch over 5 epochs, plus 8.3 hours for data collection. This is a 91.4% reduction in total training time.

  2. Inference cost. Our work targets more effective behavior steering rather than inference efficiency. Compared with Reflexion, vSFT reduces context usage and steps (Fig.4) but adds overhead for generating and loading LoRAs.

    On ALFWorld's seen split, using a single RTX 4090 (48 GB) GPU, Reflexion takes 591.8s, while vSFT takes 607.7s (2.7% longer). LoRA generation and loading account for 60.9s (10.0%) and 4.1s (0.7%), respectively. The inference cost breakdown is presented in Table R7. Note that vSFT spends less time than Reflexion on agent inference and environment interaction by completing tasks in fewer steps, and on reflection and skill generation by leveraging feedback more effectively (Fig. 2), thereby reducing the number of supervisor calls.

Table R7: Inference cost breakdown on ALFWorld's seen split. We evaluate Reflexion and vSFT on ALFWorld's seen split (140) tasks using a single RTX 4090 (48GB) GPU. All models are served using vLLM.

ComponentReflexionvSFT
Agent inference and environment interaction457.63s (77.33%)421.45s (69.35%)
Reflection and skill generation101.47s (17.15%)83.43s (13.73%)
LoRA generation and exportN/A60.93s (10.03%)
LoRA loadingN/A4.08s (0.67%)
Agent initialization28.23s (4.77%)20.27s (3.34%)
Hypernetwork initializationN/A12.50s (2.06%)
Others4.45s (0.75%)5.01s (0.82%)
Total591.79s607.68s

Comparisons at matched test-time attempts

In Table R1, we compare vSFT against GRPO and GiGPO on ALFWorld under matched test-time attempts. We also apply the hypernetwork trained for the base model directly to the GiGPO model. These experiments reveal two advantages of vSFT over RL:

  1. vSFT is complementary to RL. Although we compare vSFT with RL, it is not limited to base models and can also be applied to RL models. Notably, our hypernetwork trained on the base model can still effectively improve its GiGPO model, yielding greater gains than in-context reflection (i.e., Reflexion). Thus, GRPO and GiGPO are complementary to, rather than direct alternatives to, vSFT; Reflexion is a more direct baseline.

  2. vSFT outperforms GRPO/GiGPO under matched attempts. With fewer test-time attempts per task, vSFT still substantially outperforms both GRPO and GiGPO. Given the observed performance saturation of RL models, we do not allocate more attempts.

Table R1. Comparisons under matched test-time attempts on ALFWorld. We evaluate GRPO and GiGPO with three attempts at each of four temperatures (0, 0.1, 0.2, and 0.3). Methods with reflection are allowed 10 additional reflection trials.

MethodSeenUnseenAvg.
GRPO85.086.685.8
GiGPO87.984.386.1
GiGPO+Reflexion87.985.186.5
GiGPO+vSFT90.087.388.7
GiGPO+vSFT$^\dagger$90.790.390.5
GiGPO+vSFT$^\ddagger$90.791.090.9
vSFT92.992.592.7
vSFT$^\dagger$96.497.096.7
vSFT$^\ddagger$99.3100.099.7

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Weakness 1 & Question 1

W1 & Q1: The paper lacks comparisons with RL methods in two respects: 1. performance under matched test-time attempts, and 2. detailed training and inference costs, including hypernetwork training and additional inference overhead.

We sincerely appreciate the reviewer's careful reading and constructive suggestions. Accordingly, we conduct additional experiments and analyses to address these concerns about vSFT's advantages over RL.


Comparisons at matched test-time attempts

In Table R1, we compare vSFT against GRPO and GiGPO on ALFWorld under matched test-time attempts. We also apply the hypernetwork trained for the base model directly to the GiGPO model. These experiments reveal two advantages of vSFT over RL:

  1. vSFT is complementary to RL. Although we compare vSFT with RL, it is not limited to base models and can also be applied to RL models. Notably, our hypernetwork trained on the base model can still effectively improve its GiGPO model, yielding greater gains than in-context reflection (i.e., Reflexion). Thus, GRPO and GiGPO are complementary to, rather than direct alternatives to, vSFT; Reflexion is a more direct baseline.

  2. vSFT outperforms GRPO/GiGPO under matched attempts. With fewer test-time attempts per task, vSFT still substantially outperforms both GRPO and GiGPO. Given the observed performance saturation of RL models, we do not allocate more attempts.

Table R1. Comparisons under matched test-time attempts on ALFWorld. We evaluate GRPO and GiGPO with three attempts at each of four temperatures (0, 0.1, 0.2, and 0.3). Methods with reflection are allowed 10 additional reflection trials.

MethodSeenUnseenAvg.
GRPO85.086.685.8
GiGPO87.984.386.1
GiGPO+Reflexion87.985.186.5
GiGPO+vSFT90.087.388.7
GiGPO+vSFT$^\dagger$90.790.390.5
GiGPO+vSFT$^\ddagger$90.791.090.9
vSFT92.992.592.7
vSFT$^\dagger$96.497.096.7
vSFT$^\ddagger$99.3100.099.7

Remark. Yue et al. [1] reinterpret RL as eliciting capabilities already present in LLMs, noting that a base model can achieve higher performance by sampling. From this perspective, sampling, RL, and our vSFT can all be viewed as mechanisms for policy exploration, but they differ in how it is guided. Sampling depends on chance and is inefficient; RL shifts the distribution toward rewarded behaviors but largely fixes the policy after training. In contrast, vSFT enables adaptive test-time exploration through explicit feedback. Importantly, these approaches are not mutually exclusive and can be combined.

[1] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?


Detailed training and inference costs

As the reviewer noted, these methods present different trade-offs between training and inference costs. RL incurs substantial training costs but typically requires only one inference attempt, whereas Reflexion requires no additional training but relies on repeated test-time reflection. vSFT lies between them in training costs (higher than Reflexion but substantially lower than RL), while introducing a more complex inference procedure.

To mitigate vSFT's inference overhead, we show in Fig. 6 that its test-time adaptation on in-split tasks can serve as a form of training for held-out tasks, improving their performance without additional inference overhead on those tasks.

In this response, we provide a more detailed analysis of vSFT's training and inference costs, beyond the rollout counts reported in the original paper:

  1. Inference cost. Our work targets more effective behavior steering rather than inference efficiency. Compared with Reflexion, vSFT reduces context usage and steps (Fig.4) but adds overhead for generating and loading LoRAs.

    On ALFWorld's seen split, using a single RTX 4090 (48 GB) GPU, Reflexion takes 591.8s, while vSFT takes 607.7s (2.7% longer). LoRA generation and loading account for 60.9s (10.0%) and 4.1s (0.7%), respectively.

  2. Training cost. Compared with GRPO, vSFT has a simpler training pipeline and requires substantially less training time for two reasons:

    • Decoupled data collection. Unlike on-policy RL methods such as GRPO, which interleave rollout collection with training updates, vSFT allows offline and off-policy data collection.
    • Simpler supervised optimization. Our hypernetwork is trained with standard cross-entropy loss on previously collected successful trajectories. In contrast, GRPO estimates relative advantages from rewards across multiple rollouts and optimizes a clipped policy objective. Thus, vSFT avoids online reward evaluation, advantage estimation, and policy-ratio clipping during training.

    On ALFWorld, using a single RTX 6000D (84 GB) GPU, GRPO requires 62.3 hours per epoch over 10 epochs. In contrast, vSFT requires 9.1 hours per epoch over 5 epochs, plus 8.3 hours for data collection. This is a 91.4% reduction in total training time.

AuthorsOpenReview ↗

Response to Weakness 2 & Question 2

W2 & Q2: The source of performance gains (task-independent instructions or task-specific reflections) is unclear.

Thank you for this insightful question!

We understand the reviewer's concern that the hypernetwork may not genuinely improve the agent through reflections, but instead merely memorize or learn capabilities from training tasks and retrieve them at test time based on task information contained in the reflections.

To address this concern, we investigate whether the hypernetwork learns how to steer the model or merely stores static capabilities through the following additional experiments:

  • Structured output steering. In Table R2, we evaluate our hypernetwork on JSONSchemaBench [2], an out-of-domain testbed for verifying and comparing the effectiveness of steering model behavior. We observe that our hypernetwork outperforms both the in-context baseline and D2L, particularly on the hard split, suggesting that it can more effectively steer model behavior according to unseen requirements, rather than merely memorizing capabilities from training tasks.

  • Effects of feedback type. In Table R3, as requested by the reviewer, we examine the effects of feedback type. The results suggest that failure-independent feedback can also be helpful, but is substantially less effective than reflections that diagnose the current failure. This is intuitive: more relevant information provides more effective guidance for the agent.

  • Effects of reflection quality. In Table R4, we further investigate the impact of reflection quality using supervisor models with different capability levels. These results show that stronger supervisors (i.e., higher-quality reflections) yield progressively better performance. This further verifies that the hypernetwork does more than store static capabilities; instead, it learns to internalize external reflections into the model at test time.

Table R2. JSONSchemaBench results. Each cell reports JSON compliance/schema compliance (%). The hypernetwork is trained on ALFWorld. We internalize the system prompt and few-shot examples, while keeping the specific schema in context.

MethodGitHub EasyGitHub MediumGitHub Hard
In-context98.3/88.290.4/66.967.1/32.3
D2L98.1/85.490.6/63.164.4/30.7
vSFT (ours)99.1/91.798.1/71.086.1/37.2

Table R3. Ablation of feedback types. Results are obtained under the vSFT$^\dagger$ setting.

FeedbackALFWorld SeenALFWorld UnseenWebShop ScoreWebShop SR
No feedback (1 trial)60.764.250.6127.4
Task instruction75.077.461.1141.6
Generic guidance82.981.060.3441.0
Reflection from orther tasks88.088.964.7348.0
Reflection from current task (default)96.497.082.1959.8

Table R4. Effects of reflection quality on WebShop. Results are obtained under the vSFT$^\dagger$ setting.

SupervisorScoreSR
Qwen2.5-0.5B64.9045.0
Qwen2.5-1.5B66.3347.2
Qwen2.5-3B68.6350.2
Qwen-Flash70.7055.4
Qwen3.7-Plus70.5458.8
Qwen3.8-Max76.6162.2

[2] JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

AuthorsOpenReview ↗

Opportunity for Further Discussion

Dear Reviewer rapH,

Thank you again for your constructive review. We hope our rebuttal, particularly the additional experiments, has addressed your concerns.

As the discussion period draws to a close, we would be grateful to know whether any concerns remain and would be happy to provide further clarification.

Thank you for your time and consideration.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.