- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces OaTTT, where an LLM agent, where and LLM agent adapts to continuous task stream guided by sparse terminal verification. The authors propose ASCENT, which self-distills verified successful trajectories using LoRA. Specifically, the frozen teacher conditions on admissibility-filtered successful trajectories to provide full-vocabulary soft next-token distributions, which the student fits via forward KL divergence. Experiments across ALFWorld, WebShop, and AppWorld show that ASCENT steadily enhances task success rates and interaction efficiency while generalizing effectively to unseen environments.
Critical strengths
The paper reveals why naive token-level imitation collapses under single-rollout, sparse-reward constraints. This paper proposes an self-distillation method, using a frozen base model, achieving stable test-time adaptation without external stronger teachers.
Critical weaknesses
-
Computing full-vocabulary logits over long trajectories using the frozen teacher and executing backpropagation immediately after every successful episode introduces non-trivial inference latency and GPU overhead during deployment.
-
While soft-target distillation is introduced to stabilize single-episode updates without a replay buffer, experiments are limited to Qwen3.5-4B and 9B. The generalizability and stability of this update scheme have not been demonstrated on larger base models.
-
Parameter updates are strictly gated on episode success. On challenging tasks where a smaller or weaker model has near-zero initial task success, the agent cannot generate any privileged trajectories and the adaptation remains inactive; in such scenarios, approaches that incorporate guidance from stronger external models may be more practical.
Questions for authors
The paper demonstrates that soft-target self-distillation avoids policy collapse under single-episode updates on Qwen3.5-4B and 9B backbones. Have the authors evaluated or observed this optimization stability on larger open-weight models? Do larger models maintain comparable action-token entropy and executable formatting stability under single-rollout distillation, or do they become more susceptible to localized overfitting on episode-specific trajectories?
7 comments
W3. Learning when the base model rarely succeeds.
Thanks for the insightful review! ASCENT updates only on verified successes, so rare successes limit learning, and adaptation remains inactive if the model never succeeds anywhere in the task stream. We address this concern from two perspectives:
(a) Successful experience can transfer to rarely solved tasks. Rare success in one task family can coexist with useful successful experience elsewhere in the stream. In Table 1, ASCENT improves Heat/Cool success from 6.2%/8.0% to 20.8%/54.7% at 4B and from 0.0%/8.0% to 37.5%/48.0% at 9B. These gains are consistent with transfer across task families.
(b) Stronger-model guidance. To better understand (i) learning from tasks with rare successes: we study learning restricted to difficult Heat/Cool tasks in ALFWorld and (ii) the role of stronger-model guidance: to increase the update frequency, we also allow failed trajectories to participate in training through three variants. Direct failure updates include the failed trajectory in the privileged information, explicitly labelled unsuccessful. Self-rewriting uses the frozen policy model to rewrite the failed trajectory into partial progress and corrective guidance. DeepSeek-V4-Flash rewriting follows the same procedure using a stronger model. The rewrites retain the original failure outcome, and the proposed corrections are not executed or verified as successful.
Qwen3.5-4B
| Update setting | Update tasks | Update on failure | Heat | Cool | Overall |
|---|---|---|---|---|---|
| Frozen base | None | No | 6.2 | 8.0 | 46.4 |
| ASCENT | All tasks | No | 20.8 | 54.7 | 69.5 |
| ASCENT w/ failure updates | All tasks | Yes | 6.3 | 54.0 | 60.0 |
| ASCENT w/ self-rewritten failures | All tasks | Yes | 15.6 | 46.0 | 67.5 |
| ASCENT w/ DSv4-Flash-rewritten failures | All tasks | Yes | 18.8 | 68.0 | 75.7 |
| ASCENT | Heat/Cool tasks only | No | 0.0 | 20.0 | 47.9 |
| ASCENT w/ failure updates | Heat/Cool tasks only | Yes | 0.0 | 36.0 | 44.3 |
| ASCENT w/ self-rewritten failures | Heat/Cool tasks only | Yes | 0.0 | 44.0 | 51.4 |
| ASCENT w/ DSv4-Flash-rewritten failures | Heat/Cool tasks only | Yes | 0.0 | 64.0 | 60.0 |
Completed results are included in Table 15 of the revision.
Qwen3.5-9B
| Update setting | Update tasks | Update on failure | Heat | Cool | Overall |
|---|---|---|---|---|---|
| Frozen base | None | No | 0.0 | 8.0 | 55.0 |
| ASCENT | All tasks | No | 37.5 | 48.0 | 77.4 |
| ASCENT w/ failure updates | All tasks | Yes | 37.5 | 58.0 | 80.7 |
| ASCENT w/ self-rewritten failures | All tasks | Yes | 0.0 | 40.0 | 70.0 |
| ASCENT w/ DSv4-Flash-rewritten failures | All tasks | Yes | 52.1 | 60.0 | 82.9 |
| ASCENT | Heat/Cool tasks only | No | 25.0 | 12.0 | 57.1 |
| ASCENT w/ failure updates | Heat/Cool tasks only | Yes | 18.8 | 20.0 | 52.9 |
| ASCENT w/ self-rewritten failures | Heat/Cool tasks only | Yes | 12.5 | 28.0 | 60.0 |
| ASCENT w/ DSv4-Flash-rewritten failures | Heat/Cool tasks only | Yes | 31.3 | 60.0 | 74.3 |
Completed results are included in Table 15 of the revision.
When success-only updates are restricted to Heat/Cool, ASCENT performs only five updates at 4B and seven at 9B. Allowing success-only updates across all task families improves both Heat and Cool success at both scales. This family-specific comparison is consistent with transfer to hard tasks from experience elsewhere in the stream.
Direct failure updates and self-rewriting give mixed results in the all-task setting. Under Heat/Cool-only updates, DSv4-Flash rewriting raises Cool success from 20.0% to 64.0% at 4B and from 12.0% to 60.0% at 9B compared with success-only learning. It also outperforms self-rewriting, which likewise allows failure updates, on Cool at 4B and on both families at 9B. Heat remains unsolved at 4B. These results support stronger-model guidance as a useful complement to ASCENT when successful experience is scarce.
Thank you again for your insightful review and constructive suggestions. If our clarifications and additional experiments address your concerns, we would be grateful if you would consider raising your score. We would be happy to clarify any remaining questions.
Response by Reviewer NTX5
Thank you for the detailed responses and additional experiments.
However, I still have concern for the method’s practical applicability. When deployed, the model requires frequent training, which impose heavy computation resources. More importantly, success-only adaptation provides few learning opportunities when the base model rarely succeeds. Therefore, I will maintain my original score.
We thank the reviewer for recognizing our analysis of self-imitation collapse and stable test-time adaptation without a stronger external teacher. We address each of the concerns below.
W1. Compute and memory overhead.
Thanks for the constructive review! Test-time training indeed brings additional computation and GPU memory usage. We note that deployment latency reflects both adaptation and the interaction efficiency required to complete tasks.
Parentheses in the ASCENT columns show relative changes from the frozen base.
On ALFWorld, ASCENT improves success with around 18 additional seconds per episode. On WebShop, ASCENT more than doubles success, with lower episode time at 4B and comparable time at 9B. These results demonstrate ASCENT's practical contribution: higher task success with fewer interactions, whose time savings can offset part of the update cost (Tables 1 and 2).
At 4B/9B, the trainer and serving engine share one GPU, and the teacher reuses the trainer's frozen base weights. During development, we also tested top-128 teacher-target caching to save some GPU memory. Peak trainer GPU allocation during updates decreased from 27.6 to 16.7 GiB at 4B and from 33.0 to 25.2 GiB at 9B, excluding the serving engine, but we observed no accompanying latency reduction.Appendix C.5.2 also examines selective updates, which reduce update counts with lower observed success. We leave more effective selection rules and optimized training kernels to future work.
W2,Q1. Generalizability and optimization stability on larger models.
Generalizability. Thanks for the helpful suggestion! To examine generalizability beyond 4B/9B, we evaluated dense Qwen3.5-27B and Qwen3.5-35B-A3B (mixture of experts) on ALFWorld seen. ASCENT achieves the highest success rate and fewest interaction turns among the compared methods on both backbones.
Table 10 in the revision. Success rate (%) and mean turns on ALFWorld seen.
Stability. We extended Fig. 1 to both larger backbones under single-episode updates without replay. Fig. 8 in the revision uses the same four diagnostics: first-response-token entropy, KL to the initial policy, valid-action rate, and generated tokens per turn. ASCENT improves success and formatting without first-response-token entropy collapse.
First-response-token entropy remains near the base level at 27B and increases at 35B-A3B. Each task is scored before its own update, so the improved success and formatting on subsequent tasks support useful cross-task adaptation under single-episode updates. With adapted weights fixed, ASCENT also improves success on 134 unseen ALFWorld tasks from 43.3% to 62.7% at 4B and from 49.3% to 88.8% at 9B (Fig. 3), demonstrating transfer beyond the adaptation episodes.