acceptodds
Under review as a conference paper at ICLR 2027

Agentic-TTT: Training test-time policy for test-time training

Abstract

Test-time training (TTT) adapts an LLM's parameters using signals derived from test inputs, and can make striking improvements in pre-specified settings such as IMO competitions or designated open problems. By turning deployment experience into parameter updates, TTT provides a direct mechanism for model-level self-improvement. Yet TTT is not universally beneficial: each TTT algorithm works in different settings, and applying an ill-suited method could waste test-time compute or even damage model performance. Therefore, such parameter-level self-improvement requires agency: the model must decide when TTT is warranted, which algorithm to invoke, and whether an existing skill can be reused. To fill this gap, we introduce Agentic-TTT, which learns a test-time policy to govern those decisions. Agentic-TTT turns TTT procedures into callable tools, treats accumulated skills as an evolving deployment environment, and trains its policy using the observed utility gains from its decisions. On our benchmark, Agentic-TTT nearly doubles the utility over the backbone model, learns to trade off utility against compute, and generalizes to domains unseen during training. Together, these results point toward autonomous self-improvement: models that can decide how to learn from their own deployment experience.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Recursive self-improvement, Self-Evolving Agent, Test-time training, Agentic post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_naawOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper find that existing TTT methods work unevenly across tasks. To address this, the authors combine five existing TTT methods as callable tools and maintain a skill library. This capability is implemented by SFT and RL process. On a curated benchmark, the trained model improve performance significantly.

Critical strengths
  1. This paper identify several key problems in TTT. When to explore, when to exploit, and how to do it.
  2. This paper propose a unified framework to combine all the 5 TTT methods.
  3. Beyond final acc, the paper also considers the exact cost.
Critical weaknesses
  1. I think the main limitation of this paper is the lack of novelty. It merely combines several existing methods, without solving any new problems.
  2. The evaluation benchmark deliberately selects queries that benefit from TTT, limiting the generalizability of the proposed method.
  3. Ablation studies do not sufficiently isolate the cause of improvement. Moreover, it could be better to test what happens without skill reuse.
  4. The cost analysis does not include response time or else metrics which related to real world application. Besides, the paper should compare with methods that use the same amount of compute.
Questions for authors

Could the authors clarify the position of this paper? ( Something more than simple combination of existing methods)

6 comments
AuthorsOpenReview ↗

Response to Reviewer naaw (Split 1 of 5)

We thank the reviewer for your careful reading and constructive comments. We appreciate your recognition of the challenges we identify in TTT and our consideration of cost-utility trade-off. We address the reviewer's concerns below.

W1&Q1. The position of this paper.

W1. I think the main limitation of this paper is the lack of novelty. It merely combines several existing methods, without solving any new problems.

Q1. Could the authors clarify the position of this paper? (Something more than simple combination of existing methods)

Agentic-TTT is not just trivial combination of TTT methods. Its core is a deployment-time decision problem: how a model can autonomously decide whether to adapt, which TTT method to use, and when to reuse previously acquired capabilities for self-improvement. Our contribution lies in formulating, training and evaluating a policy for these decisions.

1. Existing TTT methods do not autonomously solve deployment-time method selection and application.

We show that TTT methods cannot be unconditionally applied or universally beneficial across all cases. They rely on various prerequisites and have unique failure modes: for example, TTT-FS [1] requires in-context demonstrations; TTRL [2] explicitly discusses failure when the model lacks prior knowledge (in Section 4.3 of their paper).

Our experiment in Table 1 further shows that fixed-method baselines produce uneven or negative benefits across domains. These findings motivate an explicit decision layer: deployment requires determining which TTT adaptation, if any, is appropriate for the given test query. Our Agentic-TTT makes this decision problem a learning objective, rather than fixed rules.

2. Agentic-TTT supports capability accumulation and reuse through stateful sequential decisions.

We view model deployment as a stateful sequential decision process over evolving sample pools and skill library. Our policy must organize adaptation material, determine when a method's prerequisites are satisfied, and identify when existing skills can serve future queries. Method selection is only one part of this broader capability-management problem: the model grows its capability set over time. This is very different from many previous TTT works, as they mostly treat each query in isolation, and focus on immediate rewards without considering long-term self-improvement.

Our additional experiments support the value of these designs. As reported in our responses to Reviewer XYgN's Q1 on routing baselines and your W3 on skill reuse, both learned routers based on Qwen3-Embedding-8B and the framework variant without skill reuse substantially underperform our framework. These results highlight the importance of management over the whole skill system.

3. Agentic-TTT is scalable to additional TTT methods.

Agentic-TTT treats all TTT methods as callable tools. The harness and policy-learning pipeline operate over this interface, separating the decision process from individual TTT algorithms. Additional future compatible TTT methods can therefore be incorporated into our framework, expanding the repertoire available for deployment-time self-improvement.


In conclusion, our work

  • Offers a new perspective: No single TTT method is universally beneficial, and a model needs wise decision-making to achieve general self-improvement.

  • Formulates a new research problem: How to achieve autonomous decision-making over callable TTT tools and evolving skill library?

  • Delivers a new methodology: an executable harness, guidance distillation followed by outcome-driven RL, and a training objective that controls the utility-computation trade-off.


Reference

[1] Akyürek, Ekin, et al. "The surprising effectiveness of test-time training for few-shot learning." International Conference on Machine Learning (ICML), 2025.

[2] Zuo, Yuxin, et al. "Ttrl: Test-time reinforcement learning." The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025).

AuthorsOpenReview ↗

Response to Reviewer naaw (Split 2 of 5)

W2. Evaluating on benchmark distribution where TTT benefits are smaller.

W2. The evaluation benchmark deliberately selects queries that benefit from TTT, limiting the generalizability of the proposed method.

We thank the reviewer for this constructive suggestion. We constructed an unfiltered benchmark and evaluated both the original Agentic-TTT checkpoint without retraining and a policy retrained on the new training distribution.


New benchmark construction: we uniformly sampled queries from the same source datasets as our original benchmark, without filtering by TTT benefits. Using the same heuristic assignment of reference TTT methods, we obtained the following distribution:

Split  Queries  Direct already correct  Reference TTT -> still wrong  Reference TTT -> beneficial 
Train2448913025 (10.2%)
Dev2449513019 (7.8%)
Eval2429212426 (10.7%)

Only 10.7% of evaluation queries benefit from reference TTT, compared with approximately 65% in the original evaluation set. These categories describe outcomes under the assigned reference method, rather than all possible TTT methods. Choosing an unsuitable TTT method can harm performance, as illustrated by the negative gains from applying SEAL to math and coding tasks (c.f. Table 1 of our paper, the "Always-SEAL" column), where the method is poorly matched to the task.


Agentic-TTT retains utility gains without retraining.

We compare utility, TTT build counts, and total evaluation FLOPs below. Compute includes TTT sampling and learning, policy reasoning, and answer generation, normalized by the direct-answer baseline’s total FLOPs (5.57×10165.57\times10^{16}).

Method   Utility  TTT Build count  FLOPs / Direct 
Direct Answer0.38001.00
Prompted Policy0.3976321.33
Semantic Router0.41320522.80
Agentic-TTT (original)0.43811624.79
Agentic-TTT (retrained)0.4268318.66
Oracle0.483338.00

Transferable benefits. The original checkpoint improves utility by 5.8 percentage points over direct answering and outperforms prompted and semantic routing baselines. This suggests that our policy trained on a benefit-enriched distribution can transfer its knowledge of suitable TTT methods to settings where beneficial opportunities are much less frequent, rather than applying TTT indiscriminately.

Behavioral differences. The training distribution also appears to shape the policy’s behavioral tendency. The original policy, trained on a benefit-enriched distribution, uses TTT more aggressively; while the policy trained on the lower-benefit distribution produces more conservative behavior. During evaluation, build operations decrease from 116 to 83, and total compute falls from 24.79 to 18.66 times the direct-answer baseline.


These results show that Agentic-TTT retains substantial utility gains on an unfiltered, lower-benefit distribution without retraining. However, the original policy may still adapt too aggressively, spending compute on attempts that do not improve outcomes. While this checkpoint remains effective, training on the distribution that reflects the expected deployment setting can help achieve a better utility–compute trade-off by encouraging a more conservative adaptation strategy. We have added this evaluation to the revised paper (Appendix B.3) and summarize it in the Introduction.

AuthorsOpenReview ↗

Response to Reviewer naaw (Split 3 of 5)

W3. Ablation study: without skill reuse.

W3. Ablation studies do not sufficiently isolate the cause of improvement. Moreover, it could be better to test what happens without skill reuse.

We thank the reviewer for these constructive comments. We examine the advantage of skill reuse through both a breakdown of the trained policy's behavior and a retrained policy variant without reuse.


Skill reuse provides substantial gains at low marginal cost. As shown in the table below, reuse accounts for only 17.1% of evaluation events and requires just 4.3 minutes in total for loading skills and answering queries. It consumes only 1.5% of the overall evaluation time while contributing 44.8% of the observed utility gain, yielding substantial benefits at low marginal cost. Once a useful skill has been built, reuse can provide an efficient and economical way to benefit future queries without repeating adaptation cost.

Action category  Triggers/evaluation (action %)  Evaluation time (time %)  Utility contribution (gain %) 
Build / Update97.3 (49.9%)251.4 min (84.9%)+0.140 (55.2%)
Reuse33.3 (17.1%)4.3 min (1.5%)+0.114 (44.8%)
Direct Answer64.3 (33.0%)5.7 min (1.9%)0.000 (0.0%)

Removing reuse increases building frequency but reduces utility. Following the reviewer’s suggestion, we retrained Agentic-TTT with skill reuse removed from its available actions:

Variant    Utility    Build / Reuse / Direct Answer count    Total evaluation time  
Agentic-TTT0.51097.3 / 33.3 / 64.3296.20 min
Without skill reuse0.439126.7 / 0 / 68.3473.15 min

Without reuse, the policy relies more heavily on new adaptations. It performs 30.2% more build operations, with evaluation time increasing from 296.2 to 473.15 minutes; yet the utility falls from 0.510 to 0.439. More frequent adaptation therefore does not compensate for the loss of skill reuse. This ablation supports the importance of reusing acquired capabilities for effective self-improvement across queries. We have added both analyses to the revised paper: Section 3.1 reports the time and utility shares of skill reuse, Section 3.5.3 presents the ablation without reuse, and Appendix F.3 provides the full breakdown.

Reviewer_EnCPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Agentic-TTT, a framework that trains an LLM-based policy to decide when and how to perform test-time training (TTT). The framework exposes multiple TTT algorithms as callable tools and maintains a persistent library of LoRA adapters that can be constructed and reused across queries. The policy is trained in two stages: guidance distillation using heuristic action sequences, followed by reinforcement learning with utility-based rewards.

Experiments cover eight domains and multiple backbone models. On Qwen2.5-7B-Instruct, the reported overall utility improves from 0.256 with direct inference to 0.510 with Agentic-TTT. The paper also studies computation-aware adaptation, transfer to unseen domains, and generalization across model families.

Critical strengths
  1. The paper addresses a relevant problem: different TTT methods have heterogeneous effectiveness and computational costs, making it desirable to learn when to invoke an adaptation method rather than applying a fixed procedure. Modeling the selection and reuse of parameterized skills as sequential decision-making is a reasonable formulation of this problem.
  2. The proposed framework demonstrates substantial improvements over direct inference and the evaluated routing baselines. The comparison between SFT-only and SFT+RL suggests that outcome-based optimization provides additional benefits beyond supervised action imitation. The experiments on computation-aware adaptation and different model families also provide useful evidence about the behavior of the framework.
Critical weaknesses
  1. The evaluation benchmark is heavily biased toward TTT-beneficial queries (65%), whereas only 7% of the original MATH-500 dataset meets the same high-gain criterion under PoT. This selection substantially favors the proposed approach and limits the generalizability of the reported improvements. Evaluation on unfiltered benchmark distributions is needed to demonstrate that the policy can reliably identify when TTT is unnecessary or harmful.
  2. During RL training, the query stream advances through oracle-annotated actions rather than the policy's own decisions (Section 2.2.2). Consequently, the policy is not directly optimized for the long-term consequences of skill construction and reuse. This training-deployment mismatch weakens the claim of autonomous self-improvement, particularly because cross-query credit assignment is central to the proposed framework.
  3. The experiments do not adequately distinguish the benefits of parameter updates from additional inference-time computation. In particular, PoT performs iterative search and can directly return the best candidate found during search. Without compute-matched baselines such as search without parameter updates or iterative self-correction, it is unclear how much of the reported improvement actually comes from TTT rather than additional search.
Questions for authors
  1. Could you evaluate Agentic-TTT on unfiltered benchmark distributions to verify its effectiveness when TTT benefits are less frequent?
  2. How does performance change when the RL training stream advances using the policy's own actions instead of oracle-annotated actions?
  3. How much improvement comes from parameter updates rather than additional search? Could you provide compute-matched search-only baselines?
4 comments
AuthorsOpenReview ↗

Response to Reviewer EnCP (Split 1 of 3)

We sincerely thank the reviewer for recognizing the relevant problem, the reasonable formulation, the substantial improvements, and the useful empirical evidence. We address the reviewer’s concerns and questions below.

W1&Q1. Evaluating on benchmark distribution where TTT benefits are smaller.

W1. Evaluation on unfiltered benchmark distributions is needed to demonstrate that the policy can reliably identify when TTT is unnecessary or harmful.

Q1. Could you evaluate Agentic-TTT on unfiltered benchmark distributions to verify its effectiveness when TTT benefits are less frequent?

We thank the reviewer for this constructive suggestion. We constructed an unfiltered benchmark and evaluated both the original Agentic-TTT checkpoint without retraining and a policy retrained on the new training distribution (Appendix B.3 of the revised paper).


New benchmark construction: we uniformly sampled queries from the same source datasets as our original benchmark, without filtering by TTT benefits. Using the same heuristic assignment of reference TTT methods, we obtained the following distribution:

Split  Queries  Direct already correct  Reference TTT -> still wrong  Reference TTT -> beneficial 
Train2448913025 (10.2%)
Dev2449513019 (7.8%)
Eval2429212426 (10.7%)

Only 10.7% of evaluation queries benefit from reference TTT, compared with approximately 65% in the original evaluation set. These categories describe outcomes under the assigned reference method, rather than all possible TTT methods. Choosing an unsuitable TTT method can harm performance, as illustrated by the negative gains from applying SEAL to math and coding tasks (c.f. Table 1, the "Always-SEAL" column), where the method is poorly matched to the task.


Agentic-TTT retains utility gains without retraining.

We compare utility, TTT build counts, and total evaluation FLOPs below. Compute includes TTT sampling and learning, policy reasoning, and answer generation, normalized by the direct-answer baseline’s total FLOPs (5.57×10165.57\times10^{16}).

Method   Utility  TTT Build count  FLOPs / Direct 
Direct Answer0.38001.00
Prompted Policy0.3976321.33
Semantic Router0.41320522.80
Agentic-TTT (original)0.43811624.79
Agentic-TTT (retrained)0.4268318.66
Oracle0.488338.00

Transferable benefits. The original checkpoint improves utility by 5.8 percentage points over direct answering and outperforms prompted and semantic routing baselines. This suggests that our policy trained on a benefit-enriched distribution can transfer its knowledge of suitable TTT methods to settings where beneficial opportunities are much less frequent, rather than applying TTT indiscriminately.

Behavioral differences. The training distribution also appears to shape the policy’s behavioral tendency. The original policy, trained on a benefit-enriched distribution, uses TTT more aggressively; while the policy trained on the lower-benefit distribution produces more conservative behavior. During evaluation, build operations decrease from 116 to 83, and total compute falls from 24.79 to 18.66 times the direct-answer baseline.


Beneficial, harmful, and outcome-preserving adaptations.

To examine the policy’s actual behavior, we compare its final answer with direct answering on queries where it invokes TTT. We report corrections and degradations relative to direct answering. Percentages are calculated over TTT-treated (build + reuse) queries within the evaluation stream.

Policy   TTT-treated queries  Incorrect → Correct  Correct → Incorrect 
Agentic-TTT (original)16816 (9.52%)2 (1.19%)
Agentic-TTT (retrained)11212 (10.71%)1 (0.89%)

This breakdown demonstrates positive net gains from adaptation even when benefits are infrequent. Agentic-TTT corrects previously wrong answers while introducing few harmful degradations.

However, TTT does not guarantee that an incorrect answer will always become correct. An attempt with a reasonable prospect of improvement may still fail, so an unchanged outcome alone does not establish that the attempt was unjustified. More accurately identifying beneficial adaptation opportunities and reducing low-value TTT operations remains an important research direction for improving compute efficiency.


These results show that Agentic-TTT retains substantial utility gains on an unfiltered, lower-benefit distribution without retraining. However, the original policy may still adapt too aggressively, spending compute on attempts that do not improve outcomes. While this checkpoint remains effective, training on the distribution that reflects the expected deployment setting can help achieve a better utility–compute trade-off by encouraging a more conservative adaptation strategy.

AuthorsOpenReview ↗

Response to Reviewer EnCP (Split 2 of 3)

W2&Q2. Policy-induced stream advancement.

W2. The policy is not directly optimized for the long-term consequences of skill construction and reuse.

Q2. How does performance change when the RL training stream advances using the policy's own actions instead of oracle-annotated actions?

We thank the reviewer for this insightful comment, which highlights a broader aspiration of our work: directly optimizing the long-term consequences of skill construction and reuse. As acknowledged in the original submission (Appendix E, Limitations), our RL stage uses a query-level proxy objective with oracle stream advancement, rather than directly optimizing returns over the complete stream.


We first examine whether changing stream advances can improve policy learning under the query-level objective. During early stages of this work, we have evaluated several policy-driven advancement schedules, inspired by the classic online learning work DAgger[1], which addresses the mismatch between expert- and self-induced training states. These schedules vary the extent to which the policy learns under states (i.e., sample pools and skill libraries) shaped by its own decisions. For each query in RL epoch ee, we define padv(e)p_{\mathrm{adv}}^{(e)} as the probability to advance the stream using policy's rather than oracle actions. All variants start from the same SFT checkpoint and undergo 33 epochs of RL training, differing only in their state advancement schedules:

Methodpadv(e)p_{\mathrm{adv}}^{(e)}Evaluated utility
Oracle-only (ours)(0,0,0)(0,0,0)0.510±0.0120.510\pm0.012
Linear oracle to policy(0.3,0.6,1)(0.3,0.6,1)0.386±0.0210.386\pm0.021
Oracle warm-up, then policy(0,1,1)(0,1,1)0.392±0.0400.392\pm0.040
Policy-only(1,1,1)(1,1,1)0.360±0.0330.360\pm0.033

All three policy-driven variants achieved utility far below oracle-only advancement, with almost minimal or even negative gains from the RL stage: as a reference, the SFT baseline has mean utility of 0.3750.375. We found that during policy-driven stream advancement, early collection and construction errors can undermine subsequent learning signals. For example, mixing incompatible samples can produce low-quality skills; collecting too few queries fails to satisfy the prerequisite for multi-query TTT adaptation. Consequently, these errors change the value and feasibility of downstream actions, making adaptation unavailable or skill reuse harmful. The resulting reward feedback obscures whether a poor outcome reflects an inappropriate current action or unfavorable states created by earlier actions. The policy may therefore learn to avoid adaptation or reuse in poorly prepared states, without learning how to establish good conditions for beneficial behaviors. These findings show that under current training objective, switching to policy-driven advancement alone cannot improve policy quality over oracle advancement.


Directly optimizing long-term cross-query returns presents following foundational challenges:

  1. Expensive long-horizon evaluation. (1) Delayed reward feedback: a complete stream-level Monte Carlo return requires waiting until the end of stream. In contrast, our query-level training provides frequent and dense feedback after each query is completed. (2) Branching states: different earlier decisions produce different sample pools and skill libraries. With kk branches per step, exhaustive expansion over nn steps yields knk^n distinct universes. Maintaining and evaluating all these branches would be costly.
  2. Uncertain reward horizon. A practical compromise is to evaluate returns over a fixed window of queries rather than the entire stream. However, a skill may be reused immediately, or after many unrelated queries, or never. The window may therefore exclude delayed benefits and wrongly evaluate actions.
  3. Causal contributions are hard to disentangle. A skill's eventual performance depends on a chain of coupled decisions. Poor sample grouping may produce a low-quality skill, yet the contribution of each query is hard to isolate. Conversely, a useful skill may yield a negative outcome when wrongly reused. Tracking these dependencies records how samples and skills were used, but does not determine how much credit or blame each decision should receive.

We view these challenges as substantial research problems. Our main contribution is to frame autonomous self-improvement as a decision process over TTT actions, and to provide a practical training approach that delivers effective decisions and empirical gains. We hope this work establishes a useful setting and inspires future work to develop better methods for long-term credit assignment and advance autonomous self-improvement.


Reference

[1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." Proceedings of the 14th international conference on artificial intelligence and statistics. 2011.

AuthorsOpenReview ↗

Response to Reviewer EnCP (Split 3 of 3)

W3&Q3. Compute-matched baselines without parameter updates.

W3. Without compute-matched baselines such as search without parameter updates or iterative self-correction, it is unclear how much of the reported improvement actually comes from TTT rather than additional search.

Q3. How much improvement comes from parameter updates rather than additional search? Could you provide compute-matched search-only baselines?

We thank the reviewer for highlighting the need to distinguish benefits of parameter updates from just scaling more test-time compute.


How PoT combines search with learning.

PoT alternates between tree search and feedback-driven learning. In the PoT original paper, search is guided by unit-test feedback for coding and reward-model scores for other domains. In our implementation, each coding query includes a subset of the unit tests. PoT returns without learning if the initial search finds a candidate passing all provided tests; otherwise, it performs reinforcement learning updates and continues searching. For non-code queries, we always perform parameter updates, using majority agreement among search candidates as feedback.

For our trained policy, PoT is in average invoked 22.3 times on code queries and 18.3 times on non-code queries per evaluation stream. The table below summarizes average counts across runs:

PoT execution path    Queries (%)    Correct w/ search+learning    Correct w/ only search  
Search only (code early exit)8.67 (21.32%)-5.00
Search + learning32.00 (78.68%)12.334.67

Thus, 78.68% of PoT calls involve parameter updates. In this subgroup, if we disable learning and perform only search, then the accuracy will drop from 38.5% to 14.6%. This ablation supports the importance of learning procedure within PoT.


FLOPs-matched baselines: test-time training vs. test-time scaling:

Following the reviewer's question, to understand whether additional inference-time computation alone can account for Agentic-TTT’s gains, we evaluate 2 baselines that keep parameters frozen while matching the overall compute. We calibrate the FLOPs of average reasoning effort per query and each TTT invocation:

Operation   Reasoning per query  PoT  TTT-FS  TTRL  SEAL  TLM 
Average FLOPs2.30×10142.30\times10^{14}1.32×10151.32\times10^{15}3.65×10163.65\times10^{16}2.35×10162.35\times10^{16}2.48×10162.48\times10^{16}5.49×10145.49\times10^{14}

Combining these costs with the invocation counts yields approximately 1.69×10181.69 \times 10^{18} FLOPs per evaluation stream, 38.75 times the reasoning-only compute cost. We implement two compute-matched baselines according to this budget:

  • Parallel test-time scaling / Best-/Majority-of-N Search:

    We independently sample NN candidate answers and select the majority answer. For coding queries, we instead use best-of-N, selecting candidates with the highest pass rate on the provided public unit tests. We evaluate up to N=40N=40:

    N   1  2  4  8  16  32  40 
    Utility0.2560.2730.3180.3410.3520.3520.358
  • Sequential test-time scaling / Self-refinement:

    Starting from an initial answer, the model repeatedly critiques its current response, generates feedback, and revises the answer. Here, NN denotes the number of refinement rounds, with N=0N=0 corresponding to the initial answer. Because refinement involves longer prompts and outputs, each iteration is more expensive, costing more than 2.5 standard reasoning passes. We therefore use N=16N=16 for compute-matched comparison:

    Refinement rounds   0  1  2  4  8  16 
    Utility0.2560.2950.3010.3130.3070.307

We found that parallel sampling improves utility with diminishing gains; while sequential self-refinement peaks at four rounds, with further iterations slightly reducing utility. Both test-time scaling strategies largely underperform Agentic-TTT, which achieves 0.510 utility. Thus, these inference-only scaling baselines fail to match Agentic-TTT's gains, supporting the efficiency and necessity of parameter updates for converting additional compute into utility gains.

Another advantage of Agentic-TTT is skill reuse, which allows adaptation experience to benefit future queries. In contrast, these search-only baselines solve each query independently: their search effort does not produce long-term skills, so similar queries require repeated computation. Our results show that our learned agency enables selective compute allocation, making more effective use of computation than the evaluated search-based baselines.

Dear Reviewer EnCP,

Thank you again for your time and effort in reviewing our paper. Given the limited time for discussion, we would appreciate it if you could let us know whether we have fully addressed your concerns; or if there are any further questions, we are happy to help resolve.

Best regards,

Author of #6118

Reviewer_XYgNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Agentic-TTT, a framework that learns when and how to apply test-time training (TTT). It trains a policy to select TTT methods, construct reusable LoRA skills, and balance performance gains against computational costs. Experiments demonstrate substantial improvements across multiple domains and model families.

Critical strengths
  1. The paper addresses an important practical challenge: deciding when TTT is beneficial and selecting appropriate adaptation methods rather than applying a fixed strategy. Also the author proposed an effective framework design. Integrating heterogeneous TTT methods with a persistent LoRA skill library provides a flexible mechanism for parameter-level adaptation and knowledge reuse.

  2. Encouraging experimental results. The approach substantially improves overall utility over the evaluated baselines, with additional experiments demonstrating compute-aware decisions and cross-domain generalization.

Critical weaknesses
  1. Insufficient comparisons with simpler learned policies. The baselines mainly include fixed TTT methods, prompting, and semantic routing. It remains unclear whether the improvements require the proposed agentic framework or could be achieved by a simpler supervised or cost-aware learned router.
Questions for authors

See the weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer XYgN (Split 1 of 2)

We thank the reviewer for the time and effort, and are glad that the reviewer recognizes the important practical challenge, the effective framework design, and the encouraging experimental results. We address the reviewer's concern below.

W1&Q1. Necessity of agentic framework, and simpler routing baselines.

W1&Q1. It remains unclear whether the improvements require the proposed agentic framework or could be achieved by a simpler supervised or cost-aware learned router.

We thank the reviewer for the constructive suggestions on learned routing baselines, as they provide an empirical basis for highlighting the necessity of our proposed agentic framework. Following this suggestion, we implemented a supervised router and a series of cost-aware routers (now included in Section 3.1 and Appendix F.1 of the revised paper):

Router interaction protocol: Both routers support multi-turn interaction, following the same protocol as our Agentic-TTT policy. At turn kk of query xtx_t, they receive the same observable information: the query xtx_t, the current state St,kS_{t,k} (sample-pool and skill-library snapshots) and the within-query history τt,<k\tau_{t,<k} containing previous actions and environment feedbacks. The router selects an action at,ka_{t,k} at turn kk. Each action candidate specifies an action type and its arguments, using the same JSON format as in our policy harness. After executing the selected action, the router receives environment feedback and selects the next action, until it issues GENERATE action or reaches the same six-turn limits as our LLM policy.

Router Architecture: We use Qwen3-Embedding-8B encoder, with a parameter size comparable to our Qwen2.5-7B policy. At each turn, we separately encode two text inputs: the query and environment snapshot (xt,St,k)(x_t, S_{t,k}), and the interaction history together with a candidate action (τt,<k,a~t,k)(\tau_{t,<k}, \tilde{a}_{t,k}). Their 4096-dimensional embeddings are then concatenated and passed to a two-layer MLP fθf_\theta, to produce a scalar score:

st,k(a~t,k)=fθ([E(xt,St,k);,E(τt,<k,a~t,k)]).s_{t,k}(\tilde{a}_{t,k}) = f_\theta\left(\left[E(x_t,S_{t,k});\\,E(\tau_{t,<k},\tilde{a}_{t,k})\right]\right).

Router Training: Both routers assign a scalar score to each candidate action. We apply a softmax over these scores and train the router with cross-entropy against the gold action. During training, the encoder is frozen and only the MLP is trained. At test time, the router selects the highest-scoring candidate action. All routers are trained for two epochs, with learning rates tuned on the dev set. They differ in how their gold actions are constructed:

  • Supervised Router: We use the oracle actions from the original SFT dataset as gold labels.

  • Cost-aware Routers: Following Section 3.2, we select gold actions based on the utility-cost net benefit ΔU−λcomplog⁡10(1+cm)\Delta U - \lambda_{\mathrm{comp}}\log_{10}(1+c_m), retaining TTT trajectories only when the utility gain pays off the computation cost. Otherwise, we use direct answering. We train one router per λcomp={0.1,0.2,0.4}\lambda_{\mathrm{comp}} = \{0.1, 0.2, 0.4\}.


(We show the results and provide analysis of these router baselines in the next split. To be continued ...)

AuthorsOpenReview ↗

Response to Reviewer XYgN (Split 2 of 2)

W1&Q1. Necessity of agentic framework, and simpler routing baselines. (Continued from last split)

W1&Q1. It remains unclear whether the improvements require the proposed agentic framework or could be achieved by a simpler supervised or cost-aware learned router.

The learned router baselines underperform Agentic-TTT across all evaluated settings:

Settings    Agentic-TTT      Router    
Supervised0.510±0.0120.510\pm0.0120.369±0.0290.369\pm0.029
Cost-aware with λcomp=0.1\lambda_{\mathrm{comp}} =0.10.479±0.0580.479\pm0.0580.356±0.0310.356\pm0.031
Cost-aware with λcomp=0.2\lambda_{\mathrm{comp}} =0.20.475±0.0350.475\pm0.0350.326±0.0260.326\pm0.026
Cost-aware with λcomp=0.4\lambda_{\mathrm{comp}} =0.40.428±0.0140.428\pm0.0140.273±0.0120.273\pm0.012

We further examined action prediction and execution behavior to understand these performance differences.


Learning and generalization: We measured turn-level gold-action agreement, which is the fraction of agreement between selected actions and gold actions, on the SFT training set and the test set:

PolicyTraining agreementTest agreement
LLM policy0.9600.689
Supervised router0.8510.430

Generalization in this setting requires handling both unseen queries and evolving skill-library states that differ from those encountered during training. The router achieves lower agreement on both splits, with a substantially larger gap on the test set. These results indicate that Agentic-TTT learns the annotated decisions more effectively and generalizes better to unseen contexts than the supervised router.


Execution failures: We also observed frequent repetition and unsuccessful recovery in the router baselines. The following table reports average occurrence counts per evaluation stream:

BehaviorLLM policyRouter
Repeated loading adapters within a query6.67374.67
Repeating a rejected action unchanged14.33113.00
Queries exhausting the six-turn budget23.46128.67

These observations show that the router baselines struggle to adapt their decisions to evolving environment states and feedback across multiple interaction turns.


Why do router baselines largely underperform our LLM-based agentic policy? We provide analysis from the following factors:

  1. Limited sensitivity to input changes.
    The routers represent context and candidate actions using 4096-dimensional embeddings. Different candidate actions or adjacent-turn environment states produced highly similar embeddings, with cosine similarities above 0.97. This may help explain the scoring head’s limited sensitivity to changes that require different decisions.

  2. Unsuccessful response to environment feedback.
    We measured successful action recovery: whether a rejected action is followed by a revised action that the environment accepts. Recovery rates were 48%–58% for the LLM policy, compared with only 1%–2% for the supervised router, demonstrating more effective use of feedback to revise decisions.

  3. Free-form reasoning and RL opportunities.
    Our LLM policy can devote additional test-time computation to free-form reasoning before selecting an action. Generated intermediate steps can support prerequisite checking, skill comparison, and interpretation of execution feedback. Outcome-driven RL then optimizes the resulting reasoning-and-action trajectories using measured returns. In our experiments, the SFT-only LLM policy and supervised router achieve comparable utility: 0.375 and 0.369, respectively. Subsequent RL improves the LLM policy to 0.510, demonstrating the benefit of outcome-driven refinement within our reasoning-and-action framework.

In conclusion, Agentic-TTT consistently outperforms the evaluated routing baselines, supporting the effectiveness of our trained reasoning-and-action policy in managing evolving skills and responding to execution feedback.

Dear Reviewer XYgN,

Thank you again for your time and effort in reviewing our paper. Given the limited time for discussion, we would appreciate it if you could let us know whether we have fully addressed your concerns; or if there are any further questions, we are happy to help resolve.

Best regards,

Author of #6118

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.