- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies test-time training for LLM agents inside one episode. authors argue that training on the raw trajectory (like aTTT) is bad supervision, because the trajectory can contain wrong beliefs and the gain of aTTT gets smaller on long tasks.
ReTTT instead lets the agent reflect every few steps and write lessons (progress, failure, subgoal, recovery) with evidence. The lessons go through rule-based screening, a small memory with Jaccard-based insert / revise / replace, and then a LoRA adapter is trained on the new lessons plus some replay.
The same adapter is used for acting and for reflecting, so the reflector also changes during the episode. There are two versions, QA style and summary style. Experiments are on ALFWorld, ScienceWorld and SWE-bench Lite with Qwen3.5-9B and Gemma4-12B. Both versions beat aTTT in all 6 settings (Table 1), up to 12.14 points.
There are also analyses on ALFWorld with Qwen: horizon split, context memory vs parametric, frozen vs adapted reflection, ablations and compute.
Critical strengths
-
I think the question is good and the direction of the results is consistent. What an agent should train on during test-time training is a real open problem, and "train on screened lessons instead of raw trajectory tokens" is a clear idea that can be tested. Both ReTTT variants are better than aTTT in all 6 model and benchmark cells in Table 1, and some gains are big (ALFWorld Qwen 51.43 to 63.57, ScienceWorld Qwen 15.00 to 26.67). This is the main reason I think the paper is worth improving.
-
The paper has the right kind of controls and is honest about cost. Context Memory (same pipeline, frozen model, Fig 3a) and Frozen Reflection (reflection with the initial adapter, Fig 3b) are the comparisons I would want to ask for. Table 5 also reports that ReTTT is around 1.085x and 1.124x the worker time of aTTT, and the text says clearly it is not cheaper overall. This makes the comparison with aTTT more fair than many TTT papers.
Critical weaknesses
-
The statistics do not support the claims, and I think the "three seeds" claim needs explanation. Table 1 says results are averaged over three seeds, but every success rate in the paper is an exact multiple of 1/N for a single pass: ALFWorld numbers are k/140 (63.57 = 89/140, 51.43 = 72/140), ScienceWorld k/120 (26.67 = 32/120), SWE-bench k/60 (43.33 = 26/60). I checked all numbers in Table 1, Table 4 and Fig 3. Means over three independent runs should mostly be multiples of 1/(3N), so this looks like one run, or seeds that give the same output. The step and error counts in Table 3 are also integers, and the totals equal 140 times the per-task averages in Table 2 (for example 4028 = 140 x 28.77). If decoding is greedy and the seeds only change LoRA initialization, please say so, since then the three runs do not measure variance. There is no std, CI or test anywhere, but the text still says "significant impact" (Sec 3.5). Many gaps are only a few tasks: SWE-bench Qwen ReTTT vs aTTT is 26 vs 24 of 60, the Short-task gain of "5.6 points" in Table 2 is 17/18 vs 16/18 (one task), and the 7.14 point frozen vs adapted reflection gap is 10 tasks of 140. The horizon claim also holds only for ReTTT-QA: ReTTT-Sum's gain over aTTT goes down with horizon (+11.11, +8.51, +8.00 on Short, Medium, Long in Table 2). This is consequential because the paper's claims ("consistently outperforms", "larger gains on longer-horizon tasks") are all built on these differences.
-
The claimed mechanism is not really isolated, and all analyses are on one setting. Fig 3, Tables 2 to 5 are only ALFWorld with Qwen3.5-9B. In Sec 3.3, ReTTT-QA beats Context Memory by 10.00 points, but Context Memory freezes the whole model,and its reflector is also frozen. That gap mixes "parametric vs context" with "adapted vs frozen reflection". The cleaner comparison is Frozen Reflection 56.43 vs Context Memory 53.57, which is 2.86 points (4 tasks) in a single setting. Also, the LoRA is very small (rank 4 on the final attention q_proj, Table 6) and gets 7.14 or 2.51 gradient steps per episode (Table 5), and there is no control that trains the same number of steps on shuffled or unrelated lessons, so I cannot tell if the gain comes from the lesson content. Finally, "adapted reflection derives more task-specific guidance" is shown only with selected trajectories (Fig 4 to 6), although the authors already have a classifier for generic vs task-specific entries (Table 8) that could measure this over all episodes.
Questions for authors
- What does "averaged over three seeds" mean, given that all numbers are multiples of 1/N? Please give per-seed results with std (or a CI over tasks), and the decoding temperature. If ReTTT vs aTTT stays clearly separated on ALFWorld and ScienceWorld across seeds, I would raise my rating.
- How are the screening rules and the Table 8 eviction classes defined for ScienceWorld and SWE-bench? Categories like "Generic container" and "Multi-object" look designed for ALFWorld. Were the rules written by looking at the evaluation tasks? There is no separate dev split mentioned.
- Were qTTT and aTTT run with the same LoRA setup (final q_proj, rank 4, lr 5e-4), and were they tuned? Also please give the reflection and action prompts and the SWE-bench scaffold (step limit, tools).
- In Figs 7 and 8, the Base and aTTT rows have "QA UPDATE" boxes. What are these, since Base has no updates? And why does the Base agent output only "look around" for all 100 steps in the ScienceWorld cases (Figs 7, 8, 9)? Because I want to make sure the base harness is not broken.
Dear Reviewer Pomd,
Thank you again for your thoughtful feedback and follow-up questions. We have provided additional comparisons with ParamMem and PMD, clarified the methodological differences, and addressed your concerns regarding replay and statistical significance.
We hope these additional experiments and explanations have helped resolve your concerns. If you find our responses satisfactory, we would be very grateful if you could reconsider your rating.
Thank you for your time and consideration!