- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PolicyTTT, a test-time training method that aims to retain improvements found during search in the model's policy. It uses a policy to generate rubric-building procedures, a frozen executor to produce rubrics, and a bridge loss to help the policy learn from successful search results. Experiments on seven HealthBench conversations show improvements over TTT-Discover in fresh generation.
Critical strengths
-
The proposed bridge loss is effective. The ablation results show that it plays an important role in improving policy retention.
-
The proposed method shows clear improvements over the baselines in the experiments, especially in fresh generation performance across most of the seven conversations.
Critical weaknesses
-
Limited evaluation and generalization. The method is trained on only seven conversations, with a separate policy adapted to each conversation. The main evaluation uses the same conversations as training, raising concerns about overfitting. The transfer results in Appendix D.5 show only very small improvements on unseen conversations.
-
Unclear practical value. The method requires extensive training for each individual conversation, but it is unclear why this is useful in practice. Training a separate LoRA for every new conversation seems expensive and difficult to scale. Moreover, the training relies on reference rubrics that may not be available for new conversations.
Questions for authors
-
Could the authors report the computational cost, such as GPU hours and total training time, for adapting the policy to each conversation and across all seven conversations?
-
Can the authors provide stronger evidence of generalization to unseen conversations? The improvements in Appendix D.5 are very small, which raises concerns that the policy mainly learns the specific training conversation.
-
I am not fully clear about the practical value of training a separate policy for each individual problem. Could the authors clarify the intended use case? In particular, would this require training a new LoRA adapter for every problem, or is there a more scalable approach in mind?
3 comments
Dear Reviewer,
Thank you again for your thoughtful review and valuable feedback. We hope our rebuttal has addressed your concerns. We would greatly appreciate any further feedback you may have. If any questions or concerns remain, we would be happy to discuss them further.
Thank you for your time and consideration!
Thank you for the clarification. Your response addresses the concerns I raised. I will maintain my original rating.
We thank the reviewer for the review and the practical questions. Our revised claims (C1-C2) are summarized in the general response.
Q1. "Could the authors report the computational cost, such as GPU hours and total training time?"
Both methods use 24 epochs and 12,288 policy generations. Wall-clock from the logs, one run per conversation:
Q2 / W1. "Can the authors provide stronger evidence of generalization to unseen conversations?"
Five of the seven paper conversations were not used in development. We also drew three new conversations from HealthBench, not used elsewhere in this project, and ran each method 3 times.
PolicyTTT is higher on both measures in all 9 runs, each paired with TTT-Discover's run of the same seed. We do not claim transfer of one adapted policy to other conversations: as in TTT-Discover, each run adapts to one item, and on 70 unseen conversations PolicyTTT policies change recall by −0.31 to +0.74 pp relative to the base.
Q3 / W2. "Could the authors clarify the intended use case?"
Thank you for this question. TTT-Discover showed that adapting a policy to a single hard instance, such as a mathematical bound or a GPU kernel, can find strong solutions. We follow that setting and ask which TTT method finds a good rubric when a clear reward exists; here the physician rubric provides that reward. Each item needs its own LoRA, trained offline; we do not propose this for deployment, and adapting one policy across many conversations is future work. We agree that this reward needs reference rubrics. Without them, a reference-free reward is needed, and the one we tested does not yet track criterion recovery. We will state this limitation clearly in the paper.