acceptodds
Under review as a conference paper at ICLR 2027

ReTTT: Reflection-Driven Test-Time Training via Experience-Based Parametric Memory

18 views

Abstract

LLM agents accumulate valuable task-specific information through their actions, observations, and environmental feedback during multi-turn task execution. When effectively used, this information can help agents refine subsequent decisions and improve task performance. However, the learning value of an interaction depends on its context and subsequent feedback, making it challenging to extract reliable, actionable lessons and revise them as the task unfolds. In this paper, we introduce ReTTT, which enables experience-driven self-adaptation through test-time training on lessons constructed and continually revised within an episode. ReTTT lets the agent reflect on interaction feedback, maintain the resulting lessons in an explicit memory, and learn from them by updating an episode-local LoRA adapter while keeping the backbone frozen. The adapted model then performs both subsequent actions and reflection, coupling policy adaptation with the construction of future learning experience. We evaluate ReTTT on ALFWorld, ScienceWorld, and SWE-bench Lite using two model families. ReTTT outperforms trajectory-based test-time training across all six model–benchmark settings, improving task success by up to 12.14 percentage points. A frozen-reflection comparison further shows a 7.14-point benefit from adapting the reflection policy, while trajectory analysis illustrates how adapted reflection derives more task-specific guidance from failed attempts. These findings suggest that learning to reflect on its own behavior can boost agents' performance on long-horizon tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Long-Horizon Agents, Parametric Memory, Test-Time Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mBJ6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies test-time training for LLM agents inside one episode. authors argue that training on the raw trajectory (like aTTT) is bad supervision, because the trajectory can contain wrong beliefs and the gain of aTTT gets smaller on long tasks.

ReTTT instead lets the agent reflect every few steps and write lessons (progress, failure, subgoal, recovery) with evidence. The lessons go through rule-based screening, a small memory with Jaccard-based insert / revise / replace, and then a LoRA adapter is trained on the new lessons plus some replay.

The same adapter is used for acting and for reflecting, so the reflector also changes during the episode. There are two versions, QA style and summary style. Experiments are on ALFWorld, ScienceWorld and SWE-bench Lite with Qwen3.5-9B and Gemma4-12B. Both versions beat aTTT in all 6 settings (Table 1), up to 12.14 points.

There are also analyses on ALFWorld with Qwen: horizon split, context memory vs parametric, frozen vs adapted reflection, ablations and compute.

Critical strengths
  1. I think the question is good and the direction of the results is consistent. What an agent should train on during test-time training is a real open problem, and "train on screened lessons instead of raw trajectory tokens" is a clear idea that can be tested. Both ReTTT variants are better than aTTT in all 6 model and benchmark cells in Table 1, and some gains are big (ALFWorld Qwen 51.43 to 63.57, ScienceWorld Qwen 15.00 to 26.67). This is the main reason I think the paper is worth improving.

  2. The paper has the right kind of controls and is honest about cost. Context Memory (same pipeline, frozen model, Fig 3a) and Frozen Reflection (reflection with the initial adapter, Fig 3b) are the comparisons I would want to ask for. Table 5 also reports that ReTTT is around 1.085x and 1.124x the worker time of aTTT, and the text says clearly it is not cheaper overall. This makes the comparison with aTTT more fair than many TTT papers.

Critical weaknesses
  1. The statistics do not support the claims, and I think the "three seeds" claim needs explanation. Table 1 says results are averaged over three seeds, but every success rate in the paper is an exact multiple of 1/N for a single pass: ALFWorld numbers are k/140 (63.57 = 89/140, 51.43 = 72/140), ScienceWorld k/120 (26.67 = 32/120), SWE-bench k/60 (43.33 = 26/60). I checked all numbers in Table 1, Table 4 and Fig 3. Means over three independent runs should mostly be multiples of 1/(3N), so this looks like one run, or seeds that give the same output. The step and error counts in Table 3 are also integers, and the totals equal 140 times the per-task averages in Table 2 (for example 4028 = 140 x 28.77). If decoding is greedy and the seeds only change LoRA initialization, please say so, since then the three runs do not measure variance. There is no std, CI or test anywhere, but the text still says "significant impact" (Sec 3.5). Many gaps are only a few tasks: SWE-bench Qwen ReTTT vs aTTT is 26 vs 24 of 60, the Short-task gain of "5.6 points" in Table 2 is 17/18 vs 16/18 (one task), and the 7.14 point frozen vs adapted reflection gap is 10 tasks of 140. The horizon claim also holds only for ReTTT-QA: ReTTT-Sum's gain over aTTT goes down with horizon (+11.11, +8.51, +8.00 on Short, Medium, Long in Table 2). This is consequential because the paper's claims ("consistently outperforms", "larger gains on longer-horizon tasks") are all built on these differences.

  2. The claimed mechanism is not really isolated, and all analyses are on one setting. Fig 3, Tables 2 to 5 are only ALFWorld with Qwen3.5-9B. In Sec 3.3, ReTTT-QA beats Context Memory by 10.00 points, but Context Memory freezes the whole model,and its reflector is also frozen. That gap mixes "parametric vs context" with "adapted vs frozen reflection". The cleaner comparison is Frozen Reflection 56.43 vs Context Memory 53.57, which is 2.86 points (4 tasks) in a single setting. Also, the LoRA is very small (rank 4 on the final attention q_proj, Table 6) and gets 7.14 or 2.51 gradient steps per episode (Table 5), and there is no control that trains the same number of steps on shuffled or unrelated lessons, so I cannot tell if the gain comes from the lesson content. Finally, "adapted reflection derives more task-specific guidance" is shown only with selected trajectories (Fig 4 to 6), although the authors already have a classifier for generic vs task-specific entries (Table 8) that could measure this over all episodes.

Questions for authors
  1. What does "averaged over three seeds" mean, given that all numbers are multiples of 1/N? Please give per-seed results with std (or a CI over tasks), and the decoding temperature. If ReTTT vs aTTT stays clearly separated on ALFWorld and ScienceWorld across seeds, I would raise my rating.
  2. How are the screening rules and the Table 8 eviction classes defined for ScienceWorld and SWE-bench? Categories like "Generic container" and "Multi-object" look designed for ALFWorld. Were the rules written by looking at the evaluation tasks? There is no separate dev split mentioned.
  3. Were qTTT and aTTT run with the same LoRA setup (final q_proj, rank 4, lr 5e-4), and were they tuned? Also please give the reflection and action prompts and the SWE-bench scaffold (step limit, tools).
  4. In Figs 7 and 8, the Base and aTTT rows have "QA UPDATE" boxes. What are these, since Base has no updates? And why does the Base agent output only "look around" for all 100 steps in the ScienceWorld cases (Figs 7, 8, 9)? Because I want to make sure the base harness is not broken.
Reviewer_PomdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a method called ReTTT, which enables experience-driven self-adaptation through test-time training on lessons constructed and continually revised within an episode. Essentially, an agent reflects on interaction feedback and then updates a LoRA adapter by learning from an explicit memory containing those reflections. Experiments show that ReTTT is better than trajectory-based test-time training across different benchmarks. An ablation study further shows the importance of different implementation components, including replay, candidate screening, and dynamic maintenance.

Critical strengths
  1. The ReTTT achieves much better performance than baselines in 3 different tasks and for 2 different base models. This demonstrates the robustness of the method across different tasks and model families. The ablation study is also useful for understanding the causal effects of different components in the method, despite it has a problem that I will discuss more in the weaknesses section.

  2. The paper writing is good, making it clear and easy to follow. The figures are also well made and effectively illustrate the key ideas and results.

Critical weaknesses
  1. The most critical weakness is the novelty. ReTTT has no meaningful difference from ParamMem / Procedural Memory Distillation. Procedural Memory Distillation was recently released, so it counts as concurrent work, but ParamMem was released in Feb 2026, so you need to detail the key differences between your work and theirs. In your related work, you say the difference is in-episode procedural internalization, but can you clarify why ParamMem or Procedural Memory Distillation isn't in-episode procedural internalization? Are they not in-episode or not procedural? Also, why is in-episode a benefit over these works? Additionally, since these 2 works are closely related, your experiments should include them as baselines to help reviewers understand ReTTT's advantages over them.

[1] Yao, Tianjun, et al. "ParamMem: Augmenting Language Agents with Parametric Reflective Memory." arXiv preprint arXiv:2602.23320 (2026).

[2] Liu, Ye, et al. "Procedural Memory Distillation: Online Reflection for Self-Improving Language Models." arXiv preprint arXiv:2607.01480 (2026).

  1. Another issue is the ablation study: since the difference between w/o replay and the full method is very small (~1%), is this statistically significant? What is the variance here if you have multiple runs? Can you report it as score ± variance so readers can tell whether this improvement is within noise?
Questions for authors

Questions are included in the critical weaknesses section.

1 comment

Dear Reviewer Pomd,

Thank you again for your thoughtful feedback and follow-up questions. We have provided additional comparisons with ParamMem and PMD, clarified the methodological differences, and addressed your concerns regarding replay and statistical significance.

We hope these additional experiments and explanations have helped resolve your concerns. If you find our responses satisfactory, we would be very grateful if you could reconsider your rating.

Thank you for your time and consideration!

Reviewer_ScS3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReTTT, which converts agent interaction trajectories into screened and dynamically maintained procedural experience and uses test-time training to update an episode-local LoRA adapter. The adapted model generates both actions and subsequent reflections, forming a loop between experience construction and policy adaptation. The authors report improvements over trajectory-based test-time training across three interactive benchmarks and two model families.

Critical strengths

Distinguishing “recording what happened” from “extracting what should be learned for future decisions” provides a useful perspective on online adaptation during interaction. Experiments span multiple domains and models, with both ReTTT variants outperforming aTTT across all six model benchmark settings, demonstrating consistent gains.

Critical weaknesses
  1. Several key figures contain missing or misaligned arrows, misplaced icons, and unclear legends, suggesting automated preparation without adequate manual review. These issues obscure the methodological workflow and make the work difficult to understand.
  2. Replacing or deleting memory entries does not undo previous gradient updates. The paper does not verify whether the adapter retains and follows incorrect rules after the underlying experience is invalidated.
  3. Revision in the appendix uses template-based concatenation, while lexical similarity cannot reliably distinguish contradictory, synonymous, and complementary experiences. Conflicting content may therefore be internalized together.
  4. Context Memory freezes the model, whereas ReTTT adapts both action and reflection policies. This comparison cannot isolate gains from the memory representation from those due to changes in the policies and subsequent experience quality.
  5. Some examples do not establish that adaptation improves key behaviors. In Figure 7, the crucial focus action occurs before the first periodic reflection and therefore cannot be attributed to parameter updates following reflection within that episode.
Questions for authors
  1. Fix the misaligned arrows, misplaced icons, and unclear legends in the figures.
  2. Clarify whether the adapter retains and follows invalidated rules after the corresponding memory entries are replaced or deleted, and whether resetting the adapter before updating on the current memory was considered.
  3. Explain how lexical similarity and template-based concatenation avoid merging and internalizing contradictory experiences during revision.
  4. Isolate the effect of the memory representation from policy adaptation when comparing Context Memory and ReTTT (for example, by comparing Context Memory directly against Frozen Reflection).
  5. Explain why the focus action at step 3 in Figure 7 is attributed to adaptation when the first periodic reflection does not occur until step 5.
  6. Clarify the methodological contribution beyond combining reflective prompting and rule-based lexical filtering with standard LoRA updates, and explain how these hand-crafted rules generalize across different domains.
1 comment

Dear Reviewer ScS3,

Thank you again for your constructive feedback. We have carefully addressed your concerns and provided additional experimental evidence and clarifications regarding memory consistency, parametric adaptation, and methodological novelty.

We hope our responses have helped clarify the contributions and effectiveness of ReTTT. If you find that your main concerns have been adequately addressed, we would sincerely appreciate your reconsideration of the rating.

Thank you for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.