acceptodds
Under review as a conference paper at ICLR 2027

ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience

Abstract

A large language model (LLM) agent solves long-horizon tasks through many reasoning-action turns, with one verification signal at termination. Deployed agents face streams of related tasks, making their own trajectories a natural resource for improvement. In-context evolving agents store reflections, memories, or skills as text, so reuse depends on retrieving the right experience and on a frozen policy executing it. We study Online Agentic Test-Time Training (OaTTT), which trains the LLM during deployment on its own execution trajectories. Each task is attempted once, and the executed trajectory with its verification result is the only learning data for updates that persist to later tasks. Directly imitating the sampled tokens of this single rollout destabilizes the policy. We introduce ASCENT (Agentic Self-distillation for Cross-task EvolutioN at Test-time), which instead self-distills verified experience. A frozen copy of the LLM receives the verifier-accepted trajectory, with environment-rejected turns removed, as privileged information and predicts next-token distributions along it with this hindsight; distilling them into persistent LoRA fast weights updates the agent for later tasks, without an external reference solution or stronger teacher. We characterize the population target of this distillation and the limits of sparse outcome selection. Across ALFWorld, WebShop, and AppWorld on varied model scales, ASCENT improves task success and interaction efficiency as experience accumulates, outperforms the compared online adaptation methods, and transfers to held-out scenes, showing that a deployed agent can consolidate its own verified experience into its policy without a separate training phase or memory retrieval at inference.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM agents, test-time training, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NTX5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OaTTT, where an LLM agent, where and LLM agent adapts to continuous task stream guided by sparse terminal verification. The authors propose ASCENT, which self-distills verified successful trajectories using LoRA. Specifically, the frozen teacher conditions on admissibility-filtered successful trajectories to provide full-vocabulary soft next-token distributions, which the student fits via forward KL divergence. Experiments across ALFWorld, WebShop, and AppWorld show that ASCENT steadily enhances task success rates and interaction efficiency while generalizing effectively to unseen environments.

Critical strengths

The paper reveals why naive token-level imitation collapses under single-rollout, sparse-reward constraints. This paper proposes an self-distillation method, using a frozen base model, achieving stable test-time adaptation without external stronger teachers.

Critical weaknesses
  1. Computing full-vocabulary logits over long trajectories using the frozen teacher and executing backpropagation immediately after every successful episode introduces non-trivial inference latency and GPU overhead during deployment.

  2. While soft-target distillation is introduced to stabilize single-episode updates without a replay buffer, experiments are limited to Qwen3.5-4B and 9B. The generalizability and stability of this update scheme have not been demonstrated on larger base models.

  3. Parameter updates are strictly gated on episode success. On challenging tasks where a smaller or weaker model has near-zero initial task success, the agent cannot generate any privileged trajectories and the adaptation remains inactive; in such scenarios, approaches that incorporate guidance from stronger external models may be more practical.

Questions for authors

The paper demonstrates that soft-target self-distillation avoids policy collapse under single-episode updates on Qwen3.5-4B and 9B backbones. Have the authors evaluated or observed this optimization stability on larger open-weight models? Do larger models maintain comparable action-token entropy and executable formatting stability under single-rollout distillation, or do they become more susceptible to localized overfitting on episode-specific trajectories?

7 comments

We thank the reviewer for recognizing our analysis of self-imitation collapse and stable test-time adaptation without a stronger external teacher. We address each of the concerns below.

W1. Compute and memory overhead.

Thanks for the constructive review! Test-time training indeed brings additional computation and GPU memory usage. We note that deployment latency reflects both adaptation and the interaction efficiency required to complete tasks.

BenchmarkBackboneFrozen base SR (%)ASCENT SR (%)Frozen base Time (s/episode)ASCENT Time (s/episode)
ALFWorld4B46.469.5 (+49.8%)97115 (+18.6%)
ALFWorld9B55.077.4 (+40.7%)115133 (+15.7%)
WebShop4B17.841.9 (+135.4%)8776 (-12.6%)
WebShop9B17.241.5 (+141.3%)166164 (-1.2%)

Parentheses in the ASCENT columns show relative changes from the frozen base.

On ALFWorld, ASCENT improves success with around 18 additional seconds per episode. On WebShop, ASCENT more than doubles success, with lower episode time at 4B and comparable time at 9B. These results demonstrate ASCENT's practical contribution: higher task success with fewer interactions, whose time savings can offset part of the update cost (Tables 1 and 2).

At 4B/9B, the trainer and serving engine share one GPU, and the teacher reuses the trainer's frozen base weights. During development, we also tested top-128 teacher-target caching to save some GPU memory. Peak trainer GPU allocation during updates decreased from 27.6 to 16.7 GiB at 4B and from 33.0 to 25.2 GiB at 9B, excluding the serving engine, but we observed no accompanying latency reduction.Appendix C.5.2 also examines selective updates, which reduce update counts with lower observed success. We leave more effective selection rules and optimized training kernels to future work.


W2,Q1. Generalizability and optimization stability on larger models.

Generalizability. Thanks for the helpful suggestion! To examine generalizability beyond 4B/9B, we evaluated dense Qwen3.5-27B and Qwen3.5-35B-A3B (mixture of experts) on ALFWorld seen. ASCENT achieves the highest success rate and fewest interaction turns among the compared methods on both backbones.

Method27B SRTurns35B-A3B SRTurns
Frozen base74.326.470.727.5
ReasoningBank59.332.181.422.8
MemP78.626.579.325.1
TT-OPSD80.721.377.125.7
ASCENT92.118.888.816.2

Table 10 in the revision. Success rate (%) and mean turns on ALFWorld seen.

Stability. We extended Fig. 1 to both larger backbones under single-episode updates without replay. Fig. 8 in the revision uses the same four diagnostics: first-response-token entropy, KL to the initial policy, valid-action rate, and generated tokens per turn. ASCENT improves success and formatting without first-response-token entropy collapse.

BackboneMethodSuccess (%)Syntax failures (%)First-response-token entropy
27BFrozen base69.327.71.0189
27BASCENT92.114.31.0013
35B-A3BFrozen base71.422.80.6891
35B-A3BASCENT88.810.91.4415

First-response-token entropy remains near the base level at 27B and increases at 35B-A3B. Each task is scored before its own update, so the improved success and formatting on subsequent tasks support useful cross-task adaptation under single-episode updates. With adapted weights fixed, ASCENT also improves success on 134 unseen ALFWorld tasks from 43.3% to 62.7% at 4B and from 49.3% to 88.8% at 9B (Fig. 3), demonstrating transfer beyond the adaptation episodes.

W3. Learning when the base model rarely succeeds.

Thanks for the insightful review! ASCENT updates only on verified successes, so rare successes limit learning, and adaptation remains inactive if the model never succeeds anywhere in the task stream. We address this concern from two perspectives:

(a) Successful experience can transfer to rarely solved tasks. Rare success in one task family can coexist with useful successful experience elsewhere in the stream. In Table 1, ASCENT improves Heat/Cool success from 6.2%/8.0% to 20.8%/54.7% at 4B and from 0.0%/8.0% to 37.5%/48.0% at 9B. These gains are consistent with transfer across task families.

(b) Stronger-model guidance. To better understand (i) learning from tasks with rare successes: we study learning restricted to difficult Heat/Cool tasks in ALFWorld and (ii) the role of stronger-model guidance: to increase the update frequency, we also allow failed trajectories to participate in training through three variants. Direct failure updates include the failed trajectory in the privileged information, explicitly labelled unsuccessful. Self-rewriting uses the frozen policy model to rewrite the failed trajectory into partial progress and corrective guidance. DeepSeek-V4-Flash rewriting follows the same procedure using a stronger model. The rewrites retain the original failure outcome, and the proposed corrections are not executed or verified as successful.

Qwen3.5-4B

Update settingUpdate tasksUpdate on failureHeatCoolOverall
Frozen baseNoneNo6.28.046.4
ASCENTAll tasksNo20.854.769.5
ASCENT w/ failure updatesAll tasksYes6.354.060.0
ASCENT w/ self-rewritten failuresAll tasksYes15.646.067.5
ASCENT w/ DSv4-Flash-rewritten failuresAll tasksYes18.868.075.7
ASCENTHeat/Cool tasks onlyNo0.020.047.9
ASCENT w/ failure updatesHeat/Cool tasks onlyYes0.036.044.3
ASCENT w/ self-rewritten failuresHeat/Cool tasks onlyYes0.044.051.4
ASCENT w/ DSv4-Flash-rewritten failuresHeat/Cool tasks onlyYes0.064.060.0

Completed results are included in Table 15 of the revision.

Qwen3.5-9B

Update settingUpdate tasksUpdate on failureHeatCoolOverall
Frozen baseNoneNo0.08.055.0
ASCENTAll tasksNo37.548.077.4
ASCENT w/ failure updatesAll tasksYes37.558.080.7
ASCENT w/ self-rewritten failuresAll tasksYes0.040.070.0
ASCENT w/ DSv4-Flash-rewritten failuresAll tasksYes52.160.082.9
ASCENTHeat/Cool tasks onlyNo25.012.057.1
ASCENT w/ failure updatesHeat/Cool tasks onlyYes18.820.052.9
ASCENT w/ self-rewritten failuresHeat/Cool tasks onlyYes12.528.060.0
ASCENT w/ DSv4-Flash-rewritten failuresHeat/Cool tasks onlyYes31.360.074.3

Completed results are included in Table 15 of the revision.

When success-only updates are restricted to Heat/Cool, ASCENT performs only five updates at 4B and seven at 9B. Allowing success-only updates across all task families improves both Heat and Cool success at both scales. This family-specific comparison is consistent with transfer to hard tasks from experience elsewhere in the stream.

Direct failure updates and self-rewriting give mixed results in the all-task setting. Under Heat/Cool-only updates, DSv4-Flash rewriting raises Cool success from 20.0% to 64.0% at 4B and from 12.0% to 60.0% at 9B compared with success-only learning. It also outperforms self-rewriting, which likewise allows failure updates, on Cool at 4B and on both families at 9B. Heat remains unsolved at 4B. These results support stronger-model guidance as a useful complement to ASCENT when successful experience is scarce.


Thank you again for your insightful review and constructive suggestions. If our clarifications and additional experiments address your concerns, we would be grateful if you would consider raising your score. We would be happy to clarify any remaining questions.

Reviewer_NTX5OpenReview ↗

Response by Reviewer NTX5

Thank you for the detailed responses and additional experiments.

However, I still have concern for the method’s practical applicability. When deployed, the model requires frequent training, which impose heavy computation resources. More importantly, success-only adaptation provides few learning opportunities when the base model rarely succeeds. Therefore, I will maintain my original score.

Reviewer_feWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ASCENT for online test-time training of LLM agents in a single-pass task stream. When an episode passes terminal verification, ASCENT removes environment-rejected turns from the trajectory, feeds the remaining student reasoning and actions as a privileged prompt into a frozen copy of the initial model, and distills its token probabilities into persistent LoRA weights via forward KL divergence. Experiments on ALFWorld, WebShop, and AppWorld report gains over frozen and online adaptation baselines.

Critical strengths

The paper targets a practical replay-free online agent adaptation setting and demonstrates consistent empirical improvements over the untrained base model across three benchmarks.

Critical weaknesses
  1. The paper's central claim—that conditioning the teacher on the full accepted trajectory provides multi-turn hindsight credit assignment—is contradicted by its own ablation study. Because the teacher prompt (Equation 13) includes the student's verbatim reasoning and action for the exact turn being distilled, the teacher can trivially copy the student's current-turn tokens from its context rather than inferring credit from future steps. Table 3 exposes this flaw: when the student's verbatim reasoning is removed and the teacher is only given the sequence of verified actions, success on Qwen3.5-9B collapses from 77.4% to 48.6%, performing significantly worse than the untrained Frozen ReAct baseline (55.0%). This indicates that the empirical gains rely on in-context token copying rather than genuine trajectory-level hindsight.
  2. The methodological novelty is marginal, and the baseline comparisons are confounded by unfair data filtering. ASCENT simply combines existing privileged self-distillation (OPSD/SDFT) with a heuristic filter that drops environment-rejected turns, wrapped in unnecessarily heavy notation without a single overview figure in the main text. More importantly, this heuristic filter is withheld from the baselines. In Tables 1 and 2, the primary baseline (TT-OPSD) is forced to retain rejected turns and raw observations (Appendix B.2), even though Table 3 shows these two factors alone degrade ASCENT by 9.5% and 5.0% on the 4B model. Likewise, the self-imitation baselines in Section 3.2 (Online RFT+KL, Equations 5 and 6) are trained on all turns without filtering out rejected actions. Consequently, the claimed superiority over existing self-distillation and RL baselines is an artifact of crippling the baselines rather than an algorithmic advance.
  3. The evaluation on WebShop violates the binary verification formulation claimed in Section 3.1. While strict benchmark success on WebShop requires a score of 1.0, Appendix B.1 reveals that ASCENT triggers training updates using a relaxed continuous reward threshold of 0.9, without disclosing what fraction of updates come from suboptimal trajectories or whether competing baselines were granted the same relaxed threshold.
Questions for authors
  1. What is the performance of ASCENT when the student's verbatim response for the current turn is masked out from the teacher's privileged prompt during that turn's distillation?
  2. How do TT-OPSD (Tables 1 and 2) and Online RFT+KL (Figure 1) perform when they are granted the exact same rejected-turn filter and observation removal used by ASCENT?
  3. On WebShop, were all baselines also updated at the 0.9 reward threshold, and how does ASCENT perform when updates are strictly gated on a score of 1.0?
4 comments

We thank the reviewer for recognizing the practical value of our replay-free online adaptation setting and the consistent improvements over the base model across three benchmarks. We address each of the concerns below.

W1,Q1. Current-turn copying.

We clarify that ASCENT intentionally uses the complete verified trajectory, including the current turn, as privileged information for the frozen teacher. The student fits the teacher's next-token distributions at its original prefixes. Episode verification selects experience for learning, and we do not explicitly assign turn-level credit from future steps.

We note that Table 3 removes reasoning from all turns while retaining their actions, including the current action. The weaker action-only result at 9B shows the value of reasoning in the teacher's context. This ablation changes reasoning throughout the trajectory and cannot attribute the gain specifically to copying the current turn.

To answer the question directly, we remove the current turn's verbatim reasoning and action from the teacher's privileged information, leaving the student prefixes unchanged (Appendix C.3.1). We also compare with TT-OPSD, whose teacher receives only the current turn generation and returned observation.

MethodQwen3.5-4BQwen3.5-9B
Frozen base46.455.0
TT-OPSD51.061.2
ASCENT w/o current-turn information56.473.3
ASCENT69.577.4

Success rate (%) on ALFWorld seen. Fig. 9 in the revision reports the current-turn ablation.

Without current-turn information in the privileged context, ASCENT retains 43% of its improvement over the frozen base at 4B and 82% at 9B. It also exceeds TT-OPSD at both scales, although TT-OPSD includes the current response in its privileged context. Useful adaptation therefore persists without explicit current-turn information.


W2,Q2. Novelty and fair comparisons.

Thanks for the constructive review! ASCENT shares technical foundations with the broader literature on on-policy self-distillation, including OPSD and SDFT. ASCENT addresses how a deployed agent can turn its own verified, locally noisy experience into useful supervision for subsequent tasks. We highlight the following connected contributions:

  1. Constructing supervision from long-horizon interactions. In single-turn formulations of on-policy self-distillation, the model produces an answer to a fixed question. Agentic tasks involve successive reasoning–action–observation turns, where each action changes the environment and the information available for later decisions. This setting motivates three connected aspects of our design:

    (a) Sparse verification with locally noisy experience. Episode-level verification selects experience for learning and leaves the contribution of individual decisions unknown. Successful episodes can still contain rejected actions, mistaken reasoning, and unnecessary turns. Fine-grained credit assignment for individual turns remains a direction for future work.

    (b) Supervision from the agent's own experience. We note that the privileged information comes entirely from the student's interactions with the environment. The verified trajectory supplies the frozen teacher's hindsight, without an external reference solution or stronger teacher. Its soft predictions supervise updates that persist to later tasks.

    (c) Filtering with minimal overhead. ASCENT filters invalid-action turns from the teacher's privileged information using existing execution feedback. The filter changes only the teacher's context. The student still trains on all nonempty responses, including invalid-action responses. This aims to improve the learning signal from self-generated, locally noisy trajectories with sparse verification, without additional model-based post-processing.

  2. Learning during test-time deployment. We focus on Online Agentic Test-Time Training and enable online weight adaptation at per-episode wall-clock times comparable to the evaluated in-context adaptation methods (Table 1). ASCENT uses one attempt per task and a few LoRA optimization steps after each accepted episode, without a heavy post-training stage. Updated weights carry forward, and the evolved student generates experience for subsequent supervision. Fig. 2 provides an overview of this self-evolution loop in the main text. We study stability, task success, interaction length, and adaptation cost under this limited update budget.

The submission already includes ASCENT with an unfiltered reasoning-action-observation context (Table 3). It achieves 62.9%/77.9% success at 4B/9B, compared with TT-OPSD's 51.0%/61.2%. ASCENT's advantage therefore persists when its teacher also receives observations and invalid-action turns, without action-validity filtering.

To further examine filtering and observation removal, we apply both changes to TT-OPSD's privileged context. Online RFT+KL has no privileged teacher context, so we remove invalid-action responses from its likelihood and KL losses, while observations remain in the interaction history.

MethodQwen3.5-4BQwen3.5-9B
TT-OPSD51.061.2
TT-OPSD w/ filter & obs removal53.357.9
Online RFT+KL41.954.0
Online RFT+KL w/ loss filtering51.256.4
ASCENT69.577.4

Table 13 in the revision. Success rate (%) on ALFWorld seen.

These techniques improve Online RFT+KL at both scales and TT-OPSD at 4B, while TT-OPSD declines at 9B. ASCENT achieves higher success in these comparisons.


W3,Q3. WebShop verification and update thresholds.

For WebShop, we clarify that all reported WebShop success rates use the exact-success criterion (score 1.01.0), and ASCENT uses v=1[score≥0.9]v=\mathbf{1}[\text{score}\ge0.9], a binary acceptance decision for adaptation only. Across the reported experiments, only 1.73% of updates have 0.9≤score<1.00.9\le\text{score}<1.0. For the compared baselines, we use the same gate or follow the update rules specified by their original methods.

To examine the effect of this threshold, we reran ASCENT-4B and ASCENT-9B with a strict 1.01.0 update gate.

BackboneUpdate thresholdStrict success (%)Mean task score (×100)
Qwen3.5-4BScore ≥0.9\ge 0.941.962.4
Qwen3.5-4BScore =1.0=1.039.660.8
Qwen3.5-9BScore ≥0.9\ge 0.941.556.8
Qwen3.5-9BScore =1.0=1.043.259.2

Updated strict-gate results for Table 14 in the revision.

The strict-gate results are slightly lower at 4B and higher at 9B than reported in Table 2. Both exceed the strongest reported baseline at the corresponding scale, MemP at 36.8% and TT-OPSD at 24.2%. ASCENT's success advantage persists when updates require exact successes.


Thank you again for your careful review and thoughtful feedback. If our clarifications and additional experiments address your concerns, we would be grateful if you would consider raising your score. We would be happy to clarify any remaining questions.

Dear Reviewer,

We hope this message finds you well. Thank you again for your careful review and thoughtful feedback. In our rebuttal, we clarified ASCENT’s contributions and the role of privileged teacher information. We also added current-turn removal ablations, filtering controls for the baselines, and experiments requiring exact success for WebShop updates.

As the discussion period nears its end, with around one day remaining, we would sincerely appreciate it if you could revisit our responses and share whether these clarifications and additional experiments have satisfactorily addressed your concerns. If any points remain unresolved or you have further feedback, please let us know. We would be happy to discuss them further.

Thank you again for your time and effort in reviewing our paper.

Dear Reviewer,

As the discussion period ends in a few hours, we would sincerely appreciate it if you could revisit our responses and additional experiments, and share whether they address your concerns. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Reviewer_zuyqOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper applies test-time training to agentic setup: in online evaluation with streaming tasks, after each task completes, the model is trained with on-policy self-distillation to learn from successful episode. Results on several agentic benchmarks show improvement over in-context only adaption/memory methods.

Critical strengths
  1. Test-time adaption of language agents is important for scaling self-evolving AI, and this paper provides a nice method to improve adaptation through parametric learning.

  2. Results are improved over baselines and they analyzed why simple online SFT/RFT would not work, providing deep insights into their method design.

Critical weaknesses
  1. For a strict online evolving setup, the agent should not have access to ground-truth verifier. It would be great if the author can show results using LLM-as-a-Judge to decide which episode to train on.

  2. The proposed method incurs additional latency which is not analyzed in detail. See my question 3 below.

  3. The evaluated benchmarks ALFWorld and WebShop are relatively short-horizon tasks (<50 steps). Evaluation on longer horizon tasks (e.g., a subset of SWE-Bench-Verified) would greatly strengthen the paper's experiments.

Questions for authors
  1. Is there any specific reason that the parameter adaptation only happens between episode, but not within an episode? for example, an agent could update its parameter at step t in the middle of an episode.

  2. What is the training framework you used for on-policy self-distillation?

  3. Can you give more details on how you measure the wall clock time for each method? For OaTTT, does that also include the overheads like loading lora adapters? Any potential solution to reduce the latency for OaTTT, e.g. via selective, less frequent update?

5 comments

We thank the reviewer for recognizing the value of parametric adaptation for self-evolving agents, our improvements over baselines, and the insights from our analysis of online SFT/RFT. We address each of the concerns below.

W1. LLM-based verification.

Thanks for the insightful review! We clarify that ASCENT uses verification to select episodes for adaptation. This decision can come from an LLM judge, while the benchmark verifier independently measures task success. We tested both a stronger judge and the frozen backbone as a self-judge.

Update gateQwen3.5-4BQwen3.5-9B
Environment verification69.577.4
Frozen self-judge62.674.8
DeepSeek-V4-Flash67.975.7

Fig. 10 in the revision. Success rate (%) on ALFWorld seen.

We find that LLM-based verification works at both scales, with performance depending on the judge, the frozen 4B self-judge performing slightly worse, and others comparable to environment verification. These results support the feasibility of LLM-based verification under this setup, without access to environment verifications.


W2,Q3. Adaptation latency and update frequency.

(a) Thanks for the constructive feedback! For the frozen and online methods, we record wall-clock time within each program's episode loop after model initialization. We then compute the average over all episodes, including failures. Frozen base timing covers generation and environment interaction. Online memory methods additionally include episode-end memory construction, while parametric methods include their online updates. Offline training costs are reported separately in GPU-hours.

Wall-clock measurement is detailed in Appendix B.3 and Table 4 of the revision.

Time (s/episode)Qwen3.5-4BQwen3.5-9B
Total115133
Update49.439.6
↳ Teacher forward8.97.3
↳ Student optimization and update overhead40.532.3
Rollout65.693.4

Adapter synchronization is measured separately from the reported ALFWorld episode totals and adds only 0.16 s per update at 4B and 0.19 s at 9B, a negligible cost. The 9B model generates fewer turns per update (12.0 compared with 16.2 at 4B), leading to its lower update time.

(b) Less frequent and selective updates. We tested updating after every second or fourth accepted episode and selecting episodes by entropy uncertainty. For uncertainty selection, we update when an accepted episode's mean action-token entropy, computed from renormalized top-20 probabilities, is at least the median over the most recent ten accepted episodes.

Update strategyUpdatesSuccess (%)Mean turns (all)Time (s/episode)
Frozen base046.435.197
ASCENT (default)—69.526.3115
Every accepted episode9870.026.4117.12 (+1.8%)
Every second accepted episode4159.330.9105.64 (-8.1%)
Every fourth accepted episode1852.134.7101.68 (-11.6%)
Selective update via entropy2954.332.7102.03 (-11.3%)

Table 17 in the revision. Qwen3.5-4B on ALFWorld seen. Parentheses show time differences relative to the paper reported time of ASCENT.

We find that fewer scheduled updates reduce runtime latency, but lower success and longer interactions limit further episode-time savings. Entropy selection also substantially reduces updates with lower success. We leave more effective selection by novelty or expected learning benefit to future work.

W3. Longer-horizon evaluation.

Thanks for the constructive review! Within the limited time of rebuttal period, we extended our evaluation to the more challenging SWE-bench Verified benchmark, testing Qwen3.5-9B on 100 randomly selected tasks with a 100-turn budget per task.

MethodSR (%)Mean turns (successful)Mean turns (all)
Frozen base43.067.477.5
MemP44.067.476.8
ReasoningBank41.064.679.9
TT-OPSD43.065.877.7
ASCENT44.055.174.6

Table 11 in the revision. Qwen3.5-9B on SWE-bench Verified.

We observe no consistent success-rate gain from adaptation, which may reflect limited cross-task transfer. Both contextual memory and weight updates depend on prior experience being useful for later tasks. During development on ALFWorld and WebShop, we found recurring procedures that transfer naturally, such as heating objects with a microwave or checking product attributes before purchase. SWE-bench debugging tasks can require substantially different repository-specific knowledge: fixing symbolic simplification may offer little direct guidance for a plotting-library bug. Such differences may make shared patterns harder to extract through either in-context or parametric adaptation.

We also note that ASCENT uses fewer turns on successful tasks, reducing mean interaction length from 67.4 to 55.1 turns (18.2% fewer). This is consistent with Section 4.3 and Appendix C.5.1 of the paper, where ASCENT also reduces turns on ALFWorld tasks solved by both the adapted agent and the frozen base.

A promising future research direction is to study whether explicitly abstracting successful experiences into reusable information provides more transferable supervision for ASCENT’s parameter updates with raw trajectories.


Q1. Between-episode updates.

Thanks for the insightful review! Our update schedule follows the available supervision: final verification becomes available when an episode ends. Updating then allows ASCENT to learn from the completed, verified experience for later tasks.

Within an episode, partial trajectories lack this final verification signal in our setting. Learning from them would require another objective or intermediate feedback and could introduce additional noise.

One complementary role for within-episode learning is context retention and use. Earlier information may fall outside the context window or remain available but be used unreliably. [a,b] illustrate how test-time updates to a neural memory can retain historical context. Such a mechanism could help preserve earlier context for later decisions within the same task.

At a high level, ASCENT internalizes successful experience for future tasks, and within-episode memory learning could complement it by retaining within-episode context.

[a] Titans: Learning to Memorize at Test Time

[b] In-Place Test-Time Training


Q2. Training framework.

Thanks for the thoughtful review! ASCENT uses a custom PyTorch training loop with Hugging Face Transformers and PEFT. We optimize only the LoRA parameters with AdamW and implement the forward-KL loss directly in PyTorch. The frozen teacher reuses the trainer's base weights with the adapter disabled.

Rollouts use vLLM with LoRA serving. After each update, we synchronize the adapter weights for the next task without restarting the engine or transferring base weights. In the 4B/9B configuration, the trainer and serving engine run in separate processes on one GPU, holding two base-model copies in total. We use PyTorch 2.13, Transformers 5.16, PEFT 0.19, and vLLM 0.28.


Thank you again for your time and constructive feedback. If our clarifications and additional experiments address your concerns, we would be grateful if you would consider raising your score. We would be happy to clarify any remaining questions.

Dear Reviewer,

We hope this message finds you well. Thank you again for your valuable feedback and thoughtful evaluation. In our rebuttal, we clarified the verification setting, update schedule, and implementation details. We also added LLM-judge experiments, latency and update-frequency analyses, and an evaluation on SWE-bench Verified.

As the discussion period nears its end, with around one day remaining, we would sincerely appreciate it if you could revisit our responses and share whether these clarifications and supplementary results have satisfactorily addressed your concerns. If there are any additional points or feedback you would like us to consider, please let us know. We would be happy to address any remaining questions.

Thank you again for your time and effort in reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to reviewers and the Area Chair

Dear Area Chair and Reviewers,

Thank you for your time and thoughtful evaluation. ASCENT addresses an important problem in self-evolving agents: how a deployed agent can learn from its own verified experience to improve on subsequent tasks. The reviewers recognize its practical replay-free online adaptation setting, consistent empirical improvements across three benchmarks, and insightful analysis of why direct self-imitation collapses. Building on this analysis, ASCENT enables stable test-time adaptation without a stronger external teacher, demonstrating the value of consolidating experience into persistent model updates.

During the rebuttal, we have directly addressed each reviewer's concerns through targeted experiments, clarifications, and additional analyses:

  1. Current-turn information and filtering (feWY W1–W2). Gains persist without current-turn information in the teacher's privileged context and when filtering is added to TT-OPSD and Online RFT+KL (Fig. 9, Table 13).

  2. Verification (zuyq W1, feWY W3). LLM-judge experiments support alternative update selection, and strict-success WebShop updates retain gains at both scales (Fig. 10, Table 14).

  3. Larger-model stability (NTX5 W2,Q1). The 27B and 35B-A3B evaluations show improved success and formatting without first-response-token entropy collapse (Fig. 8, Table 10).

  4. Deployment cost (zuyq W2,Q3, NTX5 W1). Timing, memory, and update-selection analyses quantify the cost–success tradeoff (Tables 4 and 17).

  5. Scarce successes (NTX5 W3). Restricted Heat/Cool learning and failure-rewriting experiments show how stronger-model guidance can complement success-only adaptation (Table 15).

  6. Longer-horizon evaluation (zuyq W3). We tested 100 SWE-bench Verified tasks, finding no clear success-rate gain from the compared adaptation methods (Table 11).

During the discussion

  1. zuyq acknowledged our thorough response and reaffirmed their positive assessment.
  2. feWY has not yet responded during the discussion.
  3. NTX5 reiterated the deployment-cost and scarce-success concerns without explaining why the additional evidence was insufficient. Our follow-up showed that, on ALFWorld, ASCENT-4B outperforms frozen 9B at the same episode time (69.5% compared with 55.0% success, both at 115 seconds). We also showed that gains persist on unseen tasks after updates stop, and that stronger-model guidance can complement ASCENT when successes are scarce. This follow-up has not yet received a response.

We respectfully ask the Area Chair to consider these results and our follow-up responses alongside the original reviews. Thank you again to all reviewers and the Area Chair for your time, careful consideration, and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.