- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether test-time adaptation through external memory can serve as a more efficient alternative to test-time weight updates for scientific discovery. The authors introduce Test-Time Memory Evolution (TTME), which keeps the underlying LLM weights fixed and continually distills evaluated search experience into two memory components: Guideline Memory, which stores operational lessons from execution and validity feedback, and Strategy Memory, which extracts and consolidates reusable algorithmic ideas from successful parent-child modifications. The stored strategies are subsequently selected and adapted to guide candidate generation.
TTME is evaluated within two existing test-time-training-based discovery frameworks, TTT-Discover and ThetaEvolve, on five mathematical construction tasks and three model configurations. The reported results show that TTME achieves better best-found scores than the reported TTT baselines in all evaluated settings. On TTT-Discover with Qwen3-8B, the authors additionally report substantial computational savings, with 82.8–84.6% fewer FLOPs and 47.7–58.4% lower runtime excluding evaluation. Ablations suggest that both Guideline Memory and Strategy Memory contribute to performance.
Critical strengths
-
The paper asks an important and timely question about the necessity of test-time weight updates in scientific discovery. Rather than assuming that online RL is necessary to accumulate search experience, the paper directly compares parameter adaptation with structured external-memory adaptation. This is a useful question both scientifically and practically, given the considerable computational cost of test-time training.
-
TTME appears substantially more computationally efficient while remaining competitive or better in the reported experiments. The reported 82.8–84.6% reduction in FLOPs and 47.7–58.4% reduction in runtime on TTT-Discover/Qwen3-8B are large enough to be practically meaningful. The decomposition into Guideline Memory and Strategy Memory is also sensible, and the component ablations indicate that both contribute to the final performance.
Critical weaknesses
-
The main claim that TTME “outperforms TTT” is not sufficiently supported by repeated-run statistics. The evaluated search procedures are stochastic and highly path-dependent, yet the main tables report individual best-found scores without means, standard deviations, confidence intervals, or paired-seed comparisons. This is particularly important because several improvements over TTT are numerically very small (e.g., Erdős under Qwen3-8B and gpt-oss-120b). In the ThetaEvolve experiments, Appendix B explicitly indicates that the authors' w/o-TTT and TTME conditions use only seed 3407, whereas the original inference-only configuration contains multiple seeds. Without repeated runs, it is difficult to determine whether the reported differences reflect the adaptation method or ordinary search variance. This directly affects the paper's central conclusion.
-
The comparison with TTT is not fully controlled end-to-end because the TTT numbers are primarily taken from the original papers rather than rerun alongside TTME under an identical implementation. Although the authors carefully preserve many framework-level settings, there remain implementation differences; for example, the gpt-oss-120b comparison uses the originally reported TTT result served through Tinker, whereas the authors' fixed-weight runs use vLLM. For a stochastic generative search, especially when some final score differences are very small, differences in serving/sampling implementations, seeds, or other implementation details may matter. Therefore, I find the paper's characterization of this as a “clean comparison” somewhat stronger than the experimental setup supports. A direct rerun of TTT and TTME using the same codebase, serving stack, initialization, and multiple paired seeds would substantially strengthen the paper.
Questions for authors
1.Can the authors report repeated runs for the main TTME-vs-TTT comparisons, ideally using at least 3–5 paired random seeds? In particular, what is the run-to-run variance of the best-found scores on Erdős, AC1, AC2, AC3, and Hadamard? Several of the reported improvements over TTT are small enough that this could materially change my assessment.
2.Can the authors rerun at least the Qwen3-8B TTT baseline in the same implementation used for TTME, with identical seeds, model checkpoint, inference backend, initial search state/archive, and candidate budget? If this has already been done, please clarify which results in Tables 1–2 are newly reproduced versus copied from the original papers and provide the corresponding paired comparison.
3.How sensitive is TTME to its memory design and update hyperparameters? In particular, were , the 24-guideline capacity, the 50/50 direct-vs-strategy-guided allocation, and the memory prompt templates selected using any of the five evaluation tasks? If so, how was hyperparameter selection separated from final evaluation? A clarification would help determine the extent to which the comparison with TTT reflects adaptation mechanism versus method-specific tuning.
4.For the FLOP comparison, could the authors provide the exact accounting procedure? It would be useful to know how prefill/decode FLOPs for the additional memory-maintenance, strategy-selection, and strategy-adaptation calls are calculated, and how the forward/backward FLOPs of LoRA-based TTT are estimated. The reported efficiency improvement is an important strength of the paper and deserves a more reproducible accounting.
6 comments
Q3. How sensitive is TTME to its memory design and update hyperparameters, and how were they selected?
Thank you for this question. We did not tune these hyperparameters by comparing scores on the evaluation tasks. They were set from simple design considerations, and the same values and prompt templates are used for all tasks and all models.
- Update interval (). One step often yields too few improving children to distill a strategy, so we pool two steps per update.
- Guideline capacity (24). An initial capacity of 12 evicted most new lessons in some early experiments, so we raised it to 24. This change was based on eviction, not on scores.
- 50/50 allocation. An equal split is the simplest way to use both direct and strategy-guided generation for each parent.
Our results reflect one shared configuration, and we have not yet measured sensitivity to these choices. Our claim does not rely on them being optimal. We will add a sensitivity analysis in the revision.
Q4. Provide the exact FLOP accounting procedure.
Thank you for your good suggestion. The details are listed in follows.
-
What is counted: We count candidate generation, memory-related LLM calls, and policy training. We exclude CPU evaluation of candidate programs, GPU idle time during evaluation, model loading, and queueing.
-
FLOPs: Let be the number of weights in the attention projections, MLPs, and output layer. For computed prompt tokens and output tokens, both candidate generation and memory calls cost where the attention cost at token position is approximated by ( layers, attention width ), using mean prompt and output lengths.
-
Runtime: We measure the GPU time of each component directly: the duration of each training step, generation window, and memory call, times the number of GPUs used.
Follow-up discussion
Dear Reviewer gaXA,
We greatly appreciate your thorough and insightful review, which is invaluable in improving our work. We have tried our best to carefully address your concerns: we added repeated-run statistics, reran TTT under the same implementation as TTME, and clarified how the hyperparameters were selected and how FLOPs were counted.
We hope that these additional results and explanations have adequately addressed your concerns. Nevertheless, we remain fully prepared to provide further clarification or discussion on any outstanding points. If you find that our responses have resolved your concerns, we would be deeply grateful if you would consider revising your score in support of our submission. Thank you again for your thoughtful and valuable review!
Best Regards,
Paper5675 Authors
Thank you for your constructive review. We sincerely appreciate your recognition that the paper asks an important and timely question, and that TTME is substantially more computationally efficient while remaining competitive or better. We highly value your review and have carefully addressed each of your comments below. We hope our clarifications and additional experiments help resolve your concerns.
W1&Q1. Support the main TTME-vs-TTT comparison with repeated-run statistics.
Thank you for raising the question of run-to-run variation. We agree that it matters when the differences are small, and we now report repeated runs for all three model–framework settings. Each run contributes the best score found within that run, and we report the sample mean and SD across runs.
TTME has a better mean on all three tasks and a smaller SD. On AC3, every TTME seed scored better than every RL seed.
The TTME run scored better than all three TTT reruns on AC1 and AC2, and better than two of the three on Erdős.
On Erdős, all three TTME runs improved on the TTT score. On AC2, the TTME mean matches the TTT score. On AC1, the TTME mean is slightly worse, within one SD.
W2 & Q2. Rerun TTT and TTME under the same implementation, and clarify which results in Tables 1–2 are reproduced.
Thank you for this suggestion.
Imported vs. reproduced. The TTT scores in Tables 1 and 2 are taken from the TTT-Discover and ThetaEvolve papers. All other TTT results come from our own runs: the three Qwen3-8B reruns in W1, the runtime and FLOPs in Table 3, and the analyses in Figures 4–5.
Qwen3-8B rerun under the same implementation. We reran TTT three times with the same setting as TTME. The TTME run scored better than all three TTT reruns on AC1 and AC2, and better than two of the three on Erdős (see table in W1). Under ThetaEvolve, we compare both methods at the same three seeds (see table in W1).
Why Tables 1–2 keep the published scores. At Qwen3-8B, the published TTT scores are better than the mean of our reruns on all three tasks, so comparing TTME with them is conservative. We will state these sources explicitly in the revision.