acceptodds
Under review as a conference paper at ICLR 2027

From Hesitation to Intuition: Internalizing Test-Time Adaptation into LLM Agents via On-Policy Self-Distillation

Abstract

Memory-augmented large language model (LLM) agents adapt at test time by retrieving past experience into their context, thereby avoiding the prohibitive cost of updating parameters online. Existing approaches, however, often suffer from two fundamental limitations: the decision of when to refresh the strategy relies on the agent's own self-evaluation, which is vulnerable to cognitive hallucinations, and the acquired experience remains entirely outside the model parameters, leading to repeated memory retrieval and strategy reconstruction. To address these issues, we propose TMA, a Thinking Mode-Aware agentic memory framework that consolidates test-time adaptation into the policy parameters. Specifically, we learn a linear discriminant probe on the hidden states of confident and hesitant chain-of-thought (CoT) segments, where hesitant segments exhibit typical failure patterns, and project the agent's hidden state onto the learned direction as a hesitation score. Reflection is triggered only at hesitant states, refining the global strategy with retrieved trajectory memory. We further develop an on-policy self-distillation scheme, where a teacher conditioned on privileged hindsight, i.e., successful trajectories and high-quality reflections, supervises the student on its own rollouts to jointly optimize task execution and reflection generation. This enables slow reflection to be progressively absorbed into fast single-pass decisions, reducing subsequent reflection triggering. Experiments on ALFWorld, WebShop, and Jericho demonstrate that TMA outperforms state-of-the-art memory baselines with a relative success-rate gain of up to 8.7%. After five rounds of self-distillation on ALFWorld, TMA further improves the success rate by 6.4%, while reducing reflections and token consumption per task by 92% and 28%, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Adaptation, LLM Agents, Agent Memory, Continual Learning, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KZ69OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework, TMA, addressing when an agent should update its policy and how. For the "when" question, TMA builds a linear probe that detects whether the agent hesitates during reasoning, and uses this signal as the cue for subsequent optimization. For the "how" question, TMA adopts online self-distillation to improve the model's task execution and reflection capabilities. Experimental results show that TMA outperforms related baselines on three evaluation sets.

Critical strengths
  1. Reasonable novelty. The study specifically targets the sub-problem of when to refresh the policy, and introduces a simple linear probe to determine whether the model is hesitating, thereby identifying the optimization signal.
Critical weaknesses
  1. The study still has related limitations that could be improved:
    1. The evaluation does not include multiple independent runs, so the statistical robustness of the method remains unclear.
    2. Failure-case analysis appears in the appendix, but I suggest the main text explicitly highlight in which situations the framework fails, and whether the failing parts can be improved.
    3. The paper's position is that whenever thinking (hesitation) occurs, it should be trained into the parameters. The paper only demonstrates that the training is effective; the evidence that the training is necessary remains insufficient.
    4. The paper relies mainly on Qwen models; supporting more base models would be needed to demonstrate that the framework's capability is loosely coupled with the underlying base model.
Questions for authors

Please see the points raised in the Weaknesses section.

2 comments

Thank you for your response. After considering the rebuttal, I have decided to raise my score from 2 to 3.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer KZ69,

We would like to express our gratitude once again for the time and effort you have dedicated in reviewing our paper, including the rebuttal and discussion phases. Your insightful comments are invaluable for enhancing our work. In the final version of our manuscript, we will include the results of three independent runs, a clearer discussion of TMA's limitations and the role of parameter internalization, and additional cross-model generalization experiments.

Reviewer_RV5eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TMA, a framework combining adaptive reflection with experience internalization for LLM agents. A lightweight LDA probe on hidden representations detects failure-associated reasoning patterns and triggers memory retrieval and strategy revision. On-policy self-distillation then trains task execution and reflection generation using hindsight-informed teacher supervision. Experiments on ALFWorld, WebShop, and two Jericho games, together with component ablations, memory-free evaluations, and held-out-task evaluations, report improvements in task performance and reductions in reflection frequency and inference token consumption.

Critical strengths
  1. Efficient and effective reflection triggering. The probe reuses existing hidden representations without requiring an additional LLM call for the triggering decision. In Table 6, LDA achieves 59.3% success on ALFWorld versus 47.1% for periodic triggering, while both average 13 reflections per task. This supports the value of choosing when to reflect rather than simply increasing reflection frequency.

  2. Evidence of experience internalization. With external memory and online reflection disabled, TMA-OPSD achieves 68.6% success, outperforming SFT at 61.4% and standard OPD at 64.3%. The additional improvements on held-out tasks are encouraging and suggest that the benefits extend beyond immediate access to retrieved experiences.

Critical weaknesses
  1. The probe’s supervision is insufficiently documented. The 493-sample corpus is central to the proposed trigger, but the annotating model and annotation procedure are not clearly specified. This makes it difficult to assess the reliability and reproducibility of the supervision used to identify failure-associated reasoning patterns.

  2. The scope of generalization remains unclear. The main iterative experiment repeatedly uses the same tasks. Although Figure 5(b) provides held-out-task results, it is unclear whether these tasks are also excluded from memory construction and probe training. Meanwhile, the task split used for the memory-free evaluation in Table 7 is not explicit. Consequently, the evidence for transferring internalized experience to new tasks independently of retrieval remains incomplete.

Questions for authors
  1. Which model annotated the 493 reasoning samples, and what procedure was followed to assign the confident and hesitant labels?

  2. In what respects do the unseen tasks in Figure 5(b) differ from the training tasks? Besides being excluded from distillation, were they also excluded from memory construction and probe training? This would clarify the type of generalization being evaluated.

  3. Does the memory-free evaluation in Table 7 use seen or unseen tasks? If it uses seen tasks, could the authors provide a corresponding evaluation on the same unseen tasks as Figure 5(b), with retrieval and online reflection disabled, to test whether the internalized experience transfers independently?

Reviewer_qZanOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TMA (Thinking Mode-Aware Agentic Memory), which aims to solve the unreliable self-evaluation in test-time adaptation and the prohibitive cost of updating parameters online. The main contributions include: (1) using Linear Discriminant Analysis (LDA) to detect hesitant reasoning from CoT hidden states and trigger memory retrieval and reflection when necessary. (2) proposing On-Policy Self-Distillation (OPSD), which distills successful trajectories and high-quality reflections into model parameters, rather than requiring external memory.

Critical strengths
  1. The proposed method is useful. This paper uses the lightweight LDA probe to detect reasoning patterns relevant to failure in COT hidden states. Hence, it can avoid the extra LLM-based self-evaluation. As shown in the ablations, this mechanism is better than self-evaluation in ALFWorld, which proves its actual contribution to agent performance.
  2. By on-policy self-distillations, this paper includes the experience of task execution and reflection into model parameters. The experimental studies show the success rate improvements and decrease in reflection times. At the same time, the memory-free evaluation validates that the model can keep good performance without relying the external memory.
Critical weaknesses
  1. The LDA probe training only relies on 491 COT samples. Also, the performance mainly relies on evaluation of sample-level cross-validation. Hence, it lacks cross-trajectory and cross-task evaluation. Hence, the generalization ability is under concern.
  2. The main experiments is on ALFWorld. WebShop is lacking OPSD results. Again, the generalization ability is under concern.
Questions for authors
  1. My major concern is the generation abilities. Do different folds contain samples from the same tasks or trajectories? Can the authors provide held-out evaluation based on task or trajectory?
  2. Can the authors provide the performance of Standard OPD or OPD-Evolver under the same unseen-task setting? And clarify whether unseen tasks are independent of memory construction data?
2 comments

Thanks for your response. You extra experiments help me solve my concerns, and I would like to update my score

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer qZan,

We would like to express our gratitude once again for the time and effort you have dedicated in reviewing our paper, including the rebuttal and discussion phases. Your insightful comments are invaluable for enhancing our work. In the revised version of the manuscript, we have included the results of task-level cross-validation, additional comparisons with Standard OPD and OPD-Evolver on unseen tasks, and OPSD experiments on WebShop.

Comments on the paper

10 comments
AuthorsOpenReview ↗

Response to Reviewer qZan

Dear Reviewer qZan,

Thanks for your valuable feedback, and recognizing that the LDA probe avoids extra LLM-based self-evaluation and that the memory-free evaluation supports experience internalization. For your main concern about generalization, we have stated it in following aspects: i) the probe is now validated with task-grouped cross-validation, disjoint at both the task and the trajectory level. ii) We have conducted the experiments where the unseen tasks are fully isolated from memory construction, probe fitting, and distillation. The results show that TMA outperforms both Standard OPD and OPD-Evolver on them. iii) We have also added the OPSD results on WebShop in revised manuscript. The detailed responses are listed as follows.

[W1 & Q1]: "Do different folds contain samples from the same tasks or trajectories? If so, can the authors provide a held-out evaluation at the task or trajectory level?"

In the original submission different folds contained samples from the same tasks. We have replaced the sample-level split with task-grouped 10-fold cross-validation, which is disjoint at both the task and the trajectory level. Under this setting the probe remains highly discriminative, and the proposed method still outperforms all baselines on unseen tasks that are never used for memory construction, probe fitting, or distillation.

To investigate this issue, we analyzed the task origins of the 493 annotated CoT samples and found that, among the 140 ALFWorld tasks, 94 contributed more than two samples. Furthermore, we found that, on average, CoT samples from 41 tasks appeared in both the training and validation sets during cross-validation, confirming the task overlap across folds.

In addition, we conducted an task- and trajectory-level held-out evaluation and have added a detailed description of the task-aware CoT annotation and probe cross-validation procedures in Appendix A.1. Specifically, each annotated CoT sample is associated with its originating task ID, and we perform 10-fold cross-validation to ensure that samples from the same task do not appear in both the training and validation sets.

MetricResult
OOF Accuracy89.17%
OOF AUC0.955

Under the task-level cross-validation setting, we refit the LDA probe, achieving a mean out-of-fold(OOF) accuracy of 89.17% and an AUC of 0.955.

Generalization to unseen tasks. Furthermore, we evaluate the resulting probe on unseen tasks, with the results presented below:

MethodPickLookCleanHeatCoolPick2Overall SR
No Memory81.8250.0032.2647.8342.8623.5345.52
AdaMEM87.50 ± 8.3357.41 ± 8.4948.39 ± 3.2359.42 ± 6.6468.25 ± 5.5039.22 ± 8.9960.45 ± 1.29
OPD-Evolver68.06 ± 4.8155.56 ±11.1147.31 ± 1.8662.32 ± 5.0261.90 ± 9.5237.25 ± 3.4055.72 ± 1.14
TMA (w/o OPSD)84.72 ± 6.3657.41 ± 3.2151.61 ± 5.5969.57 ± 7.5371.43 ± 4.7643.14 ± 6.7963.43 ± 1.49
TMA90.28 ± 2.4159.26 ± 8.4949.46 ± 3.7268.12 ± 2.5182.54 ± 2.7552.94 ± 5.8866.92 ± 0.86

When memory is constructed using training tasks, TMA consistently outperforms other relevant memory baselines on unseen ALFWorld tasks. Specifically, TMA (w/o OPSD) outperforms AdaMEM [1] by 2.98 points, and incorporating TMA-OPSD further improves performance by 3.49 points. These results demonstrate that TMA maintains strong generalization ability on unseen tasks even under a strict cross-task evaluation setting.

We hypothesize that the identification of confident and hesitant reasoning patterns is less dependent on task-specific characteristics, as CoT samples from the same task may not necessarily exhibit strong feature correlations, allowing the probe to capture transferable reasoning patterns rather than task-specific features. This further supports that the probe learns generalizable reasoning patterns instead of relying on task-specific characteristics.

The results of task-level cross-validation provide a rigorous assessment of the probe's generalization ability and strengthens our experimental evaluation.

[1] Zhang, Yunxiang, et al. "AdaMEM: Test-Time Adaptive Memory for Language Agents." arXiv preprint arXiv:2606.05684 (2026).

AuthorsOpenReview ↗

Response to Reviewer RV5e

Dear Reviewer RV5e,

Thank you for the careful review, and for recognizing both the value of state-aware triggering and the evidence for experience internalization. Your concerns are i) the documentation of the probe's supervision and ii the scope of generalization.

In the revised manuscript, we have fully document the annotation pipeline, including a human validation of label quality. We also conduct experiments where the unseen tasks come from ALFWorld's out-of-distribution split and are excluded from memory construction, probe training, and all distillation rounds. In addition, the memory-free evaluation is also reported on these unseen tasks. The detailed responses are listed as follows

[W1 & Q1] "Which model annotated the 493 reasoning samples, and what procedure was followed to assign the confident and hesitant labels? "

The samples were annotated by DeepSeek-V4-Flash [1]. Labels are assigned by trajectory outcome combined with predefined behavioral criteria: CoT segments that make key contributions in successful trajectories are labeled confident, and CoT segments that exhibit one of three predefined failure patterns in failed trajectories are labeled hesitant. A human check on 150 samples shows about 91% agreement with the automated labels. The full procedure is now described in Appendix A.1, the complete prompts are in Appendix C.4, and the human validation is in Table 5.

We apologize for the missing details in the original submission. We adopt a two-stage pipeline for data annotation and probe fitting:

(1) Automated CoT Annotation. We employ DeepSeek-V4-Flash[1] to extract reasoning segments from trajectories stored in memory according to the predefined criteria. Specifically, the model localizes CoT segments associated with key contributions to task completion from successful trajectories as confident samples, and labels segments exhibiting predefined failure patterns, including strategy inertia, action-repetition loops, or commonsense-prior fixation, as hesitant samples in failed trajectories.

(2) Task-Level Cross-Validation. We perform 10-fold task-level cross-validation on the annotated samples to fit and evaluate the LDA probe, ensuring that samples from the same task do not appear in both the training and validation sets.

The full procedure is now described in Appendix A.1, the complete prompts are in Appendix C.4, and the human validation is in Table 5.

[1] Xu, Anyi, et al. "Deepseek-v4: Towards highly efficient million-token context intelligence." arXiv preprint arXiv:2606.19348 (2026).

AuthorsOpenReview ↗

Response to Reviewer RV5e

[W2] "The scope of generalization remains unclear. The main iterative experiment repeatedly uses the same tasks."

In our main experiments, we perform iterative TMA-OPSD evaluations on seen ALFWorld tasks, primarily focusing on the agent's ability to reuse and internalize experiences from repeated tasks. Specifically, we examine whether the agent can progressively improve task performance on recurring tasks while reducing its reliance on costly online reflection through TMA-OPSD.

In Figure 5(b), we further provide a cross-task generalization experiment, where the model undergoes five rounds of TMA-OPSD iteration on seen tasks and is subsequently evaluated on unseen tasks. The results show that, compared to the base model, task performance improves from 61.2% to 66.4%, while the average number of reflections decreases from 19.6 to 7.32. In summary, cross-task generalization is evaluated separately under strict isolation, and three experiments consistently show that the benefits carry over to tasks never used at any stage.

We attribute this generalization to two key factors. First, TMA-OPSD internalizes transferable reasoning patterns that are not tied to specific tasks, rather than memorizing task-specific solutions. Second, Reflection Distillation improves the agent's ability to generate effective corrective reflections. The learned reflective capabilities can therefore remain useful when the agent encounters previously unseen tasks.

To further address the reviewer's concern regarding generalization, we have conducted additional unseen-task experiments, where memory is constructed exclusively from training tasks, and have included the results in Table 2 of the revised manuscript.

MethodPickLookCleanHeatCoolPick2Overall SR
No Memory81.8250.0032.2647.8342.8623.5345.52
AdaMEM87.50 ± 8.3357.41 ± 8.4948.39 ± 3.2359.42 ± 6.6468.25 ± 5.5039.22 ± 8.9960.45 ± 1.29
OPD-Evolver68.06 ± 4.8155.56 ±11.1147.31 ± 1.8662.32 ± 5.0261.90 ± 9.5237.25 ± 3.4055.72 ± 1.14
TMA (w/o OPSD)84.72 ± 6.3657.41 ± 3.2151.61 ± 5.5969.57 ± 7.5371.43 ± 4.7643.14 ± 6.7963.43 ± 1.49
TMA90.28 ± 2.4159.26 ± 8.4949.46 ± 3.7268.12 ± 2.5182.54 ± 2.7552.94 ± 5.8866.92 ± 0.86

TMA consistently outperforms other relevant memory baselines on unseen ALFWorld tasks. Specifically, TMA (w/o OPSD) outperforms AdaMEM [1] by 2.98 points, and incorporating TMA-OPSD further improves performance by 3.49 points. These results provide strong evidence that our probe identifies transferable reasoning patterns rather than relying on task-specific characteristics.

[1] Zhang, Yunxiang, et al. "AdaMEM: Test-Time Adaptive Memory for Language Agents." arXiv preprint arXiv:2606.05684 (2026).

[Q2] In what respects do the unseen tasks in Figure 5(b) differ from the training tasks? Besides being excluded from distillation, were they also excluded from memory construction and probe training?

First, we would like to emphasize that the unseen tasks are entirely independent of both memory construction and probe cross-validation. These tasks are excluded from all intermediate stages and are used exclusively for final evaluation.

In Figure 5(b), the key differences between unseen and training tasks lie in their environmental settings and task-specific requirements. Therefore, the unseen-task evaluation examines whether the execution and reflective reasoning capabilities internalized through TMA-OPSD can transfer to previously unseen task instances and environments, rather than merely memorizing solutions to previously encountered tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.