- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a verbally supervised fine-tuning framework that enables dense natural-language feedback and test-time weight updates. The framework supports on-policy, in-parameter learning from dense feedback without backpropagation, through a closed-loop interaction between an agent and a supervisor. Experiments on ALFWorld and WebShop demontrate that Qwen3-4B with the proposed framework can outperform several baseline methods.
Critical strengths
The framework enables on-policy, in-parameter learning with dense natural-language feedback without requiring backpropagation. The core idea is novel, particularly the closed-loop interaction between the agent and supervisor for test-time adaptation.
Critical weaknesses
-
Several training and evaluation details remain unclear. For example, the reported time cost is lower than that of competing methods, but differences in the number of trials and the supervisor's access to agent trajectories may affect the fairness of the comparisons. These factors require further clarification.
-
The experiments are limited to a single model family. Evaluation across different model families would help establish the generalizability of the proposed framework.
Questions for authors
How are the interaction budgets and supervisor access to agent trajectories controlled to ensure fair comparisons with baselines? Does the proposed framework generalize to other model families?
5 comments
Response to Weakness 2
W2: The experiments are limited to a single model family.
Thank you for raising this important concern.
We acknowledge that empirical evaluations or theoretical analyses across model families would further support our method. Due to the limited time during rebuttal, it is hard to conduct additional experiments with a different model family.
Nevertheless, we would like to provide a theoretical perspective on the underlying mechanism of vSFT, offering insights into why it is expected to work across different models. Moreover, we conduct out-of-domain experiments to provide evidence that vSFT learns a generalizable behavior-steering mechanism.
Neural thickets and hypernetwork. Recent work [6] suggests that task-specific experts may be densely distributed in the parameter space surrounding well-pretrained models (validated across model families, including Qwen, Llama, and OLMo3), a phenomenon termed neural thickets. This finding motivates using random weight perturbations to discover such experts. From this perspective, vSFT can be viewed as a learnable, multi-round search mechanism in parameter space. Rather than relying on random perturbations, vSFT trains a hypernetwork to iteratively generate low-rank weight updates guided by verbal feedback, thereby steering the agent toward task-specific solutions. This interpretation offers a potential explanation for why the underlying mechanism could generalize across different model families.
Generalization to new domains. To examine generalization beyond ALFWorld/WebShop, we conduct additional evaluations on JSONSchemaBench [2] and SCBench [5]. As shown in Tables R2 and R6, vSFT consistently improves JSON/schema compliance, while KV caches computed with the adapter generated by vSFT for ALFWorld even achieve a higher average score on out-of-domain tasks. These results support both vSFT's transferable behavior steering ability and the cross-domain generalization of its adapted KV representations.
Table R2. JSONSchemaBench results. Each cell reports JSON compliance/schema compliance (%). The hypernetwork is trained on ALFWorld. We internalize the system prompt and few-shot examples, while keeping the specific schema in context.
| Method | GitHub Easy | GitHub Medium | GitHub Hard |
|---|---|---|---|
| In-context | 98.3/88.2 | 90.4/66.9 | 67.1/32.3 |
| D2L | 98.1/85.4 | 90.6/63.1 | 64.4/30.7 |
| vSFT (ours) | 99.1/91.7 | 98.1/71.0 | 86.1/37.2 |
Table R6: SCBench results. Base KV denotes caches computed by the base model; LoRA KV denotes caches computed with the adapter generated by vSFT for ALFWorld. The base model performs decoding.
| Task | Base KV | LoRA KV |
|---|---|---|
| Retr.KV | 54.6 | 62.2 |
| Retr.MultiHop | 40.0 | 32.0 |
| Code.RepoQA | 53.6 | 53.0 |
| En.QA | 27.3 | 24.5 |
| Zh.QA | 3.4 | 4.0 |
| En.MultiChoice | 17.8 | 29.7 |
| Math.Find | 28.3 | 14.2 |
| ICL.ManyShot | 70.7 | 70.7 |
| En.Sum | 36.1 | 36.5 |
| Mix.Sum+NIAH (Sum) | 36.4 | 37.3 |
| Mix.Sum+NIAH (NIAH) | 65.0 | 83.6 |
| Mix.RepoQA+KV (RepoQA) | 60.2 | 54.6 |
| Mix.RepoQA+KV (KV) | 81.8 | 91.2 |
| Average | 44.3 | 45.6 (+1.4) |
[2] JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
[5] SCBench: A KV Cache-Centric Analysis of Long-Context Methods
[6] Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights
Opportunity for Further Discussion
Dear Reviewer ffTt,
Thank you again for your constructive review. We hope our rebuttal, particularly the additional experiments, has addressed your concerns.
As the discussion period draws to a close, we would be grateful to know whether any concerns remain and would be happy to provide further clarification.
Thank you for your time and consideration.
Sincerely,
The Authors
Response to Weakness 1
Thank you for your careful reading and constructive suggestions.
To address this concern, we clarify the experimental settings and conduct additional experiments to ensure fairer comparisons.
Inference budgets
All methods, including RL baselines, use the same environment interaction limits per trial: 50 steps on ALFWorld and 15 steps on WebShop. However, as the reviewer noted, RL methods use only one trial per task, whereas vSFT and Reflexion allow up to 10 trials. This presents a trade-off between training and inference costs. RL incurs substantial training costs but typically requires only one inference attempt, whereas Reflexion requires no additional training but relies on repeated test-time reflection. vSFT lies between them in training costs (higher than Reflexion but substantially lower than RL), while introducing a more complex inference procedure. We provide a wall-clock time comparison below:
Regarding vSFT's substantial training savings, we provide a detailed analysis in our response to Reviewer
rapH, and a breakdown of its inference costs in our response to ReviewerkbbX. We kindly refer the reviewer to that response and would be happy to elaborate further if needed.Fairer comparisons
In the original paper, we have made efforts to ensure fair comparisons. For example, we introduce skills to make the in-context baseline (i.e., Reflexion) as strong as possible. For the same reason, we use a carefully designed, instructive system prompt (Appendix C). These designs help isolate the gains attributable to vSFT.
Nevertheless, we thank the reviewer for identifying two remaining fairness concerns:
We therefore compare vSFT against GRPO and GiGPO on ALFWorld under matched test-time attempts in Table R1. We also allow the GiGPO model to reflect on its trajectories, in a Reflexion/vSFT way. These experiments reveal two advantages of vSFT over RL:
vSFT outperforms GRPO/GiGPO under matched attempts. With fewer test-time attempts per task, vSFT still substantially outperforms both GRPO and GiGPO. Given the observed performance saturation of RL models, we do not allocate more attempts.
vSFT is complementary to RL. Although we compare vSFT with RL, it is not limited to base models and can also be applied to RL models. Notably, our hypernetwork trained on the base model can still effectively improve its GiGPO model, yielding greater gains than in-context reflection (i.e., Reflexion). Thus, GRPO and GiGPO are complementary to, rather than direct alternatives to, vSFT; Reflexion is a more direct baseline.
Table R1. Comparisons under matched test-time attempts on ALFWorld. We evaluate GRPO and GiGPO with three attempts at each of four temperatures (0, 0.1, 0.2, and 0.3). Methods with reflection are allowed 10 additional reflection trials.