acceptodds
Under review as a conference paper at ICLR 2027

PolicyTTT: The Policy as the Product of Test-Time Training for Clinical Rubric Generation

Abstract

Test-time training (TTT) improves solution discovery by updating a model during search at test time. However, methods such as TTT-Discover are primarily evaluated by the best solution found, leaving unclear whether the adapted policy itself retains these gains. We study this question in clinical rubric generation and find that discovering a high-quality rubric does not necessarily enable the adapted policy to reproduce its quality when generating anew from the same conversation alone. We introduce PolicyTTT, which explicitly trains the policy to retain improvements found during search. The policy generates rubric-construction procedures that a frozen executor follows to produce rubrics. Its key component is a bridge that trains the policy to generate successfully refined procedures from the conversation alone, without search history or evaluation feedback, connecting search-time improvements to fresh generation after adaptation. Across seven HealthBench themes, with one adaptation conversation per theme, PolicyTTT improves fresh-generation criterion recall over the base policy on all seven conversations and over TTT-Discover on six. Improvements also extend to search itself: PolicyTTT matches or exceeds TTT-Discover's best discovered rubric on six conversations. On the communication conversation, removing the bridge reduces fresh recall from to and the best discovered rubric's from to . These results show that discovering a strong solution does not by itself ensure that search gains are retained in the policy, and that explicitly training for retention can improve both fresh generation and search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Training, Rubric Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5B9NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PolicyTTT, a test-time training method that aims to retain improvements found during search in the model's policy. It uses a policy to generate rubric-building procedures, a frozen executor to produce rubrics, and a bridge loss to help the policy learn from successful search results. Experiments on seven HealthBench conversations show improvements over TTT-Discover in fresh generation.

Critical strengths
  1. The proposed bridge loss is effective. The ablation results show that it plays an important role in improving policy retention.

  2. The proposed method shows clear improvements over the baselines in the experiments, especially in fresh generation performance across most of the seven conversations.

Critical weaknesses
  1. Limited evaluation and generalization. The method is trained on only seven conversations, with a separate policy adapted to each conversation. The main evaluation uses the same conversations as training, raising concerns about overfitting. The transfer results in Appendix D.5 show only very small improvements on unseen conversations.

  2. Unclear practical value. The method requires extensive training for each individual conversation, but it is unclear why this is useful in practice. Training a separate LoRA for every new conversation seems expensive and difficult to scale. Moreover, the training relies on reference rubrics that may not be available for new conversations.

Questions for authors
  1. Could the authors report the computational cost, such as GPU hours and total training time, for adapting the policy to each conversation and across all seven conversations?

  2. Can the authors provide stronger evidence of generalization to unseen conversations? The improvements in Appendix D.5 are very small, which raises concerns that the policy mainly learns the specific training conversation.

  3. I am not fully clear about the practical value of training a separate policy for each individual problem. Could the authors clarify the intended use case? In particular, would this require training a new LoRA adapter for every problem, or is there a more scalable approach in mind?

3 comments

We thank the reviewer for the review and the practical questions. Our revised claims (C1-C2) are summarized in the general response.

Q1. "Could the authors report the computational cost, such as GPU hours and total training time?"

Both methods use 24 epochs and 12,288 policy generations. Wall-clock from the logs, one run per conversation:

TTT-DiscoverPolicyTTT
Wall-clock per epoch4.9 min10.6 min (2.2×)
Wall-clock per conversation1.96 h4.26 h
Training GPU-h per conversation (8 × H100)15.734.1
Training GPU-h, 7 conversations110238
Epochs to reach the same quality (mean)14.17.7
Wall-clock to reach the same quality (mean / median)1.33 h / 1.41 h1.45 h / 1.47 h
  • PolicyTTT is 2.2× slower per epoch because each procedure is run by the executor, but it reaches the same quality in about half the epochs, so in comparable time. Same quality is the lower of the two methods' final Best F2.
  • RFT after search adds about 5 minutes.

Q2 / W1. "Can the authors provide stronger evidence of generalization to unseen conversations?"

Five of the seven paper conversations were not used in development. We also drew three new conversations from HealthBench, not used elsewhere in this project, and ran each method 3 times.

New conversation (theme)Best F2, TTT-DiscoverBest F2, PolicyTTTFresh, TTT-DiscoverFresh, PolicyTTT
new-1 (global health)35.249.310.618.9
new-2 (health data)37.950.012.420.3
new-3 (communication)35.857.73.033.1

PolicyTTT is higher on both measures in all 9 runs, each paired with TTT-Discover's run of the same seed. We do not claim transfer of one adapted policy to other conversations: as in TTT-Discover, each run adapts to one item, and on 70 unseen conversations PolicyTTT policies change recall by −0.31 to +0.74 pp relative to the base.

Q3 / W2. "Could the authors clarify the intended use case?"

Thank you for this question. TTT-Discover showed that adapting a policy to a single hard instance, such as a mathematical bound or a GPU kernel, can find strong solutions. We follow that setting and ask which TTT method finds a good rubric when a clear reward exists; here the physician rubric provides that reward. Each item needs its own LoRA, trained offline; we do not propose this for deployment, and adapting one policy across many conversations is future work. We agree that this reward needs reference rubrics. Without them, a reference-free reward is needed, and the one we tested does not yet track criterion recovery. We will state this limitation clearly in the paper.

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback. We hope our rebuttal has addressed your concerns. We would greatly appreciate any further feedback you may have. If any questions or concerns remain, we would be happy to discuss them further.

Thank you for your time and consideration!

Thank you for the clarification. Your response addresses the concerns I raised. I will maintain my original rating.

Reviewer_XUPGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TTT-for-discovery methods (e.g., TTT-Discover) are judged by the best solution found, but the adapted policy often can't reproduce it on its own. The authors call this the "retention gap." PolicyTTT addresses it in three ways. (1) The policy writes rubric-construction procedures rather than rubrics, and a frozen, tool-using executor turns each procedure into a rubric. (2) RL updates use TTT-Discover's entropic objective, plus credit assigned to individual procedure steps. (3) A bridge trains the policy to produce successful rewrites from the conversation alone.

Critical strengths
  1. Distilling search into a policy isn't new, but this paper is the first to measure retention directly in this setting.
  2. The bridge is simple and well motivated. It directly targets the gap they diagnose: rewrite rollouts beat conversation-only rollouts in 22 of 23 epochs (Fig. 1).
Critical weaknesses
  1. The bridge is expert iteration run inside the training loop. The obvious cheaper alternative is to run TTT-Discover, then fine-tune on its best archive entries using conversation-only prompts (rejection-sampling fine-tuning, STaR-style). Without this baseline, it isn't shown that the in-loop bridge is needed.
  2. Every ablation and every analysis of why the method works (Table 1, Figs. 1, 2, 5, 6) uses only the communication conversation. Appendix F says it was used in earlier development. The paper's central causal claim, that the bridge drives retention, is never tested on the five conversations held out from development.
  3. Each condition has one run, and the CIs only cover sampling at evaluation time. The large gaps (communication, complex responses) are probably real. The small ones (hedging, health data) and the exact sizes of all gaps can't be interpreted.
  4. The same frozen matcher supplies the training reward and the evaluation score. There is no held-out judge and no human audit of the criterion matches, so the gains may partly reflect the matcher's quirks.
Questions for authors
  1. Does the bridge help when added to TTT-Discover's direct rubric-writing setup? And why does the procedure interface perform worse than direct writing without the bridge?
  2. If you run TTT-Discover and then fine-tune on its top archive entries using conversation-only prompts (rejection-sampling fine-tuning), how close does it get to PolicyTTT? This tests whether the bridge needs to run inside the loop.
  3. Why does the interface hurt without the bridge? PolicyTTT w/o bridge (23.6%) scores below TTT-Discover (41.3%). What explains this, and does it hold on other conversations?
  4. Ablations on the held-out conversations. Can you report the bridge ablation on the five conversations not used in development?
2 comments

We thank the reviewer for the detailed review and concrete suggestions. Our revised claims (C1-C2) are summarized in the general response.

W1 / Q2. "If you run TTT-Discover and then fine-tune on its top archive entries [...], how close does it get to PolicyTTT?"

Thank you for this suggestion. We ran it as described:

  1. Rank all valid rubrics from the TTT-Discover run by F2 and remove duplicates.
  2. Keep the top k (k = PolicyTTT's bridge sample count, 122-177).
  3. Fine-tune a fresh LoRA from the base policy to write them from the conversation alone.

Same RFT without search and after PolicyTTT (run 1, 7 conversations):

MethodBest F2Fresh recall
RFT only, no search35.410.6
TTT-Discover57.725.0
TTT-Discover, then RFT57.750.6
PolicyTTT62.933.6
PolicyTTT, then RFT62.951.9
  • RFT needs search. Without it, RFT reaches only 10.6, and a STaR-style loop 17.4.
  • On top of search, RFT cannot change Best F2 but substantially improves fresh recall (25.0 → 50.6).
  • With RFT on both, fresh recall is comparable and PolicyTTT retains the better best rubric (62.9 vs 57.7). We will revise our method to PolicyTTT + RFT accordingly.

W2 / Q4. "Can you report the bridge ablation on the five conversations not used in development?"

Yes; see the last two columns of the W3 table below.

  • On 4 of 5, every run without the bridge is below every PolicyTTT run. Health data is the exception.
  • On emergency referral and three new conversations, all 12 no-bridge runs are below all PolicyTTT runs.

W3. "Each condition has one run, and the CIs only cover sampling at evaluation time."

Thank you for raising this. We reran the main comparison and bridge ablation 3 times each. Mean [min-max]:

ConversationBest F2, TTT-DiscoverBest F2, PolicyTTTFresh, TTT-DiscoverFresh, PolicyTTTFresh, PolicyTTT w/o bridge
communication87.7 [81-91]90.8 [81-100]39.6 [32-46]55.5 [50-61]21.7 [19-24]
emergency referral49.6 [49-50]70.1 [61-80]21.4 [19-26]40.6 [32-50]16.9 [15-19]
hedging35.2 [27-44]45.1 [42-47]7.4 [5-9]13.4 [10-19]6.6 [6-8]
context seeking45.5 [42-51]48.5 [44-52]19.0 [18-20]29.1 [27-32]9.7 [8-11]
health data77.4 [75-83]80.4 [75-83]45.0 [44-48]35.3 [31-42]37.7 [37-38]
complex responses62.9 [60-69]74.2 [69-77]30.5 [29-32]40.7 [32-46]15.6 [15-17]
global health47.2 [42-50]47.8 [42-60]10.4 [9-12]18.8 [13-26]5.8 [4-7]
mean57.965.324.833.316.3
  • Large gaps persist; hedging favors PolicyTTT in all runs (13.4 vs 7.4).
  • Complex responses has one PolicyTTT run (31.7) inside TTT-Discover's range (29.2-31.9).
  • On health data, TTT-Discover is higher in all 3 runs (45.0 vs 35.3).

W4. "The same frozen matcher supplies the training reward and the evaluation score."

Thank you for this point. We re-scored with two judges not used in training, GPT-6 Luna and Claude Opus 5.5. Strict uses the paper's matcher prompt; intent counts criteria making the same clinical point as matching, and was defined after seeing strict results.

MethodTraining matcherGPT-6 Luna strictOpus 5.5 strictGPT-6 Luna intentOpus 5.5 intent
Best F2 (7 conversations × 3 runs)
TTT-Discover57.95.020.132.653.8
PolicyTTT65.39.123.637.862.6
Fresh recall (7 conversations, judged samples)
TTT-Discover24.44.913.919.332.4
PolicyTTT33.54.914.323.239.1
  • PolicyTTT's best rubric is higher under every judge.

Q1. "Does the bridge help when added to TTT-Discover's direct rubric-writing setup?"

Only slightly (fresh recall, 7 conversations × 3 runs):

Policy writesWithout bridgeWith bridge
Rubrics directly (TTT-Discover)24.826.2 (+1.4)
Procedures (PolicyTTT)16.333.3 (+17.0)

The bridge helps procedures, not direct rubric writing.

Q3. "Why does the interface hurt without the bridge?"

Without the bridge, the policy does not learn to write good procedures from the conversation alone. This holds on all 7 conversations (W3 table).

  • During search, good procedures come from revising a parent with its feedback. Conversation-only procedures remain weak: 19.5 recall vs 44.9 for revisions (communication, epochs 16-23).
  • Fresh generation sees only the conversation, so fresh recall remains low.
  • The bridge trains the policy to write the successful revisions from the conversation alone. In the same setting, conversation-only recall rises to 54.1, and fresh recall on communication from 21.7 to 55.5.

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback. We hope our rebuttal has addressed your concerns. We would greatly appreciate any further feedback you may have. If any questions or concerns remain, we would be happy to discuss them further.

Thank you for your time and consideration!

Reviewer_xY4cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper studies whether improvements discovered during test-time training are retained in the adapted policy when it generates again without search history or feedback. PolicyTTT generates rubric-construction procedures that a frozen executor converts into clinical evaluation rubrics

  • main component is a supervised bridge that trains successful rewritten procedures

  • Experiments adapt separate policies to seven HealthBench conversations.

  • PolicyTTT improves fresh-generation criterion recall over TTT-Discover

Critical strengths
  • Separating the best solution discovered during search from the quality of unselected fresh generations makes the results more informative.

  • The paper examines both quantities and provides criterion-level analyses that help explain what the adapted policy retains.

Critical weaknesses
  • The main results cover one conversation per theme and one adaptation run per condition. The 512 fresh generations estimate performance conditional on an adapted checkpoint; they do not measure variation across training runs or conversations. The authors acknowledge these limitations, but they still leave uncertainty about how reliably the central findings would reproduce.

  • The bridge trains on successful model-generated procedures after removing the parent and feedback from the input. Section 6 appropriately relates this to existing search-then-distill methods. However, the experiments do not establish what is gained over a straightforward baseline that fine-tunes on selected archived procedures using the same executor and conversation-only prompts. The bridge ablation establishes its importance within PolicyTTT, but does not resolve whether the complete method offers a meaningful advantage over simpler alternatives.

Questions for authors
  • How does PolicyTTT compare with directly fine-tuning on successful archived procedures

  • Have you repeated the main comparison or bridge ablation across adaptation seeds

  • Given the small transfer gains in Appendix D.5 what concrete downstream use benefits from repeatedly generating new rubrics for the same conversation after a strong rubric has already been found

2 comments

We thank the reviewer for the thoughtful review and helpful suggestions. Our revised claims (C1-C2) are summarized in the general response.

Q1 / W2. "How does PolicyTTT compare with directly fine-tuning on successful archived procedures?"

Thank you for suggesting this baseline. We ran it in three steps:

  1. Run PolicyTTT.
  2. Take the top-k procedures from its archive (k = PolicyTTT's bridge sample count).
  3. Fine-tune a fresh LoRA from the base policy to write those procedures from the conversation alone. The same frozen executor turns them into rubrics.

We compare it with RFT on the rubrics from the same archive (PolicyTTT's run-1 archive, all 7 conversations):

MethodBest F2Fresh recall
PolicyTTT62.933.6
PolicyTTT, then fine-tune on its procedures62.937.9
PolicyTTT, then RFT on its rubrics62.951.9
  • Post-hoc fine-tuning cannot change Best F2, but it further improves fresh recall (33.6 → 37.9).
  • Fine-tuning on rubrics instead of procedures improves it further (51.9). Based on this, we will revise our final method to PolicyTTT + RFT (C2).

Q2 / W1. "Have you repeated the main comparison or bridge ablation across adaptation seeds?"

Yes. We reran the main comparison and the bridge ablation 3 times each. Mean [min-max]:

ConversationBest F2, TTT-DiscoverBest F2, PolicyTTTFresh, TTT-DiscoverFresh, PolicyTTTFresh, PolicyTTT w/o bridge
communication87.7 [81-91]90.8 [81-100]39.6 [32-46]55.5 [50-61]21.7 [19-24]
emergency referral49.6 [49-50]70.1 [61-80]21.4 [19-26]40.6 [32-50]16.9 [15-19]
hedging35.2 [27-44]45.1 [42-47]7.4 [5-9]13.4 [10-19]6.6 [6-8]
context seeking45.5 [42-51]48.5 [44-52]19.0 [18-20]29.1 [27-32]9.7 [8-11]
health data77.4 [75-83]80.4 [75-83]45.0 [44-48]35.3 [31-42]37.7 [37-38]
complex responses62.9 [60-69]74.2 [69-77]30.5 [29-32]40.7 [32-46]15.6 [15-17]
global health47.2 [42-50]47.8 [42-60]10.4 [9-12]18.8 [13-26]5.8 [4-7]
mean57.965.324.833.316.3
  • Best F2: higher by mean on all 7.
  • Fresh recall: higher than TTT-Discover on 6 of 7.
  • Bridge: removing it lowers fresh recall on 6 of 7, with every run below every run with it.
  • Across conversations: on three new conversations, PolicyTTT is higher on both measures in all 9 runs (C1).

Q3. "What concrete downstream use benefits from repeatedly generating new rubrics for the same conversation after a strong rubric has already been found?"

Thank you for this question. TTT-Discover showed that test-time training on a single hard instance can find strong solutions. Following that setting, our work asks which TTT method finds a good rubric when a clear reward exists. The best rubric is therefore the main output, and our revised main claim is about finding it (C1). We do not claim transfer to other conversations, and we will present repeated generation, such as for ensembles or candidate sets for physician review, only as a secondary use.

Dear Reviewer,

Thank you again for your thoughtful review and valuable feedback. We hope our rebuttal has addressed your concerns. We would greatly appreciate any further feedback you may have. If any questions or concerns remain, we would be happy to discuss them further.

Thank you for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive reviews. Following their suggestions, we ran 3 runs per condition, three new conversations, two held-out judges, and the requested post-hoc baselines. Based on these results, we revise our claims as follows.

PaperRevised
FocusThe retention gap: whether search gains stay in the policyFinding the best solution at test time: PolicyTTT's search outperforms TTT-Discover's
Final policyThe bridge makes the policy retain more than TTT-Discover'sAdding post-hoc RFT further improves retention; PolicyTTT + RFT becomes our full method
BridgeKey componentKept; post-hoc RFT is added after search

Terms. Best F2: best rubric found by search. Fresh recall: mean recall of 512 rubrics the adapted policy writes from the conversation alone. RFT (rejection-sampling fine-tuning): after search, fine-tune the policy on the top-scoring rubrics found.

C1. PolicyTTT finds better rubrics at test time, in every comparison we ran. Finding the best solution at test time is the goal of test-time training for discovery.

Best F2TTT-DiscoverPolicyTTT
Paper's 7 conversations (3 runs each)57.965.3
3 new conversations, unused in this project (3 runs each)36.352.3 (higher in all 9 runs)
Robustness of Best F2 (paper's 7 conversations × 3 runs)TTT-DiscoverPolicyTTTNote
Same-seed pairs won (of 21)3135 tied
Epochs to reach the common level (mean over 21 runs)15.07.2PolicyTTT earlier in 19 of 21 runs
Held-out judge settings with higher Best F2 (of 4)042 judges × strict / intent

PolicyTTT finds the better rubric on both the paper's and new conversations, and the advantage holds across seeds, in speed, and under held-out judges. (Common level: the lower of the two methods' final Best F2.)

C2. Combined with RFT, PolicyTTT gives the best final policy (paper's 7 conversations, run 1):

MethodBest F2Fresh recall
RFT only, no search35.410.6
TTT-Discover (as in its paper)57.725.0
TTT-Discover, then RFT on its best rubrics57.750.6
PolicyTTT (as in our paper)62.933.6
PolicyTTT, then RFT on its best rubrics62.951.9

RFT runs after search, so Best F2 is unchanged. Without search, RFT reaches only 10.6 fresh recall, so the final policy depends on how good the search is. We will add RFT as the final step of PolicyTTT.

We will revise the paper accordingly and add the RFT baselines to the main results. We address each reviewer's remaining questions in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.