- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that direct benchmark scores reflect both model capabilities and familiarity with task formats, output requirements, and data distributions. As a result, base models may be underestimated and gains from fine-tuning may be overstated. To address this confound, the authors propose Test-Time RL Alignment (TTRA). The first stage uses one labeled example for RL-based task-format alignment. The second stage uses benchmark questions and majority-vote-derived rewards for test-time RL adaptation, followed by evaluation. This describes the central pipeline; the code and legal applications use task-specific reward construction. On HumanEval, Qwen2.5-Coder-1.5B Base improves from 3.7% pass@1 under direct evaluation to 70.2% after TTRA, while the Instruct model changes from 70.7% to 73.4%. On MATH500, Qwen2.5-Math-1.5B Base and Instruct achieve 75.50% and 74.54%, respectively, after TTRA, substantially narrowing the gap observed under direct evaluation. The paper also studies clinical and legal tasks using MIMIC-CDM and LEXAM.
Critical strengths
The paper identifies and empirically exposes task-familiarity confounding in benchmark evaluation. The TTRA results show that direct gaps between base and fine-tuned models can partly reflect familiarity with formats and distributions rather than capability alone.
Why this matters for the overall assessment: This problem formulation has direct implications for LLM evaluation practice; even with limited algorithmic novelty, the finding and protocol could justify acceptance.
Critical weaknesses
-
-
TTRA changes the evaluated model, so the meaning of the resulting score must be defined more carefully. Updating weights on benchmark questions can adapt to templates, distributions, or hidden regularities even without labels; TTRA scores are not equivalent to original zero-shot scores.
-
Why this matters for the overall assessment: If TTRA scores are presented as uncontaminated measures of model capability, the paper has a fundamental interpretational problem. This is the most important conceptual risk to resolve before acceptance.
-
-
-
The findings about post-training gains and the applicability of TTRA remain unvalidated in agentic settings. Post-training appears to substantially improve tool use, multi-turn interaction with environments, and long-horizon planning, but the paper does not establish whether the gap between base and post-trained models would still narrow substantially after the same TTRA alignment on these tasks. Success in interactive tasks also depends on sequential decisions and environmental feedback, and it is unclear how effectively the current reward construction and alignment procedure extend to multi-turn trajectories.
-
Why this matters for the overall assessment: This limits the use of the paper’s observations as a general explanation of post-training gains. Without evidence from representative agentic tasks, the results primarily support task-familiarity effects on the evaluated benchmarks; they do not establish that post-training gains in interactive tasks also mainly reflect familiarity alignment. This limitation affects my assessment of the breadth of the conclusions and the generality of the evaluation method.
-
Questions for authors
-
Could the authors explicitly define the distinction between TTRA as a train-before-test evaluation protocol and as a test-time adaptation algorithm, and explain which scores should be reported in model comparisons or leaderboards?
-
Does the reduced Base–Instruct gap reflect format adaptation, elicitation of existing reasoning capabilities, or distribution-specific specialization? Could capability retention, language drift, and transfer to other tasks help distinguish these explanations?
-
TTRA substantially improves base-model performance on mathematics and code and reduces gap to the fine-tuned variants. Can the findings be reproduced in base models from other model families, particularly general-purpose base models without mathematics- or code-specific specialization?
-
This paper already cites Yue et al. (2025), Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. Using large-k pass@k, problem coverage, and perplexity analyses, that work finds that, in the settings studied, RLVR gains mainly reflect more efficient sampling of reasoning paths already present in the base model rather than fundamentally new reasoning patterns. The narrowing of the base–fine-tuned performance gap after TTRA is consistent with this observation. What are the central differences and additional conclusions of the present paper relative to that work? In particular, is task-familiarity alignment an empirically distinguishable mechanism from improved sampling efficiency of existing reasoning paths, or an alternative interpretation of the same phenomenon? Could the authors combine pre-/post-TTRA pass@k curves and problem-coverage analyses on matched models and benchmarks to identify evidence beyond the prior work and distinguish changes in task familiarity, sampling efficiency, and reasoning capability boundaries?
-
Post-training appears to substantially improve base-model capabilities on agentic tasks, such as tool use, multi-turn interaction with environments, and long-horizon planning. Do this paper’s findings about task familiarity and post-training gains, as well as the TTRA method itself, remain applicable and effective in these settings?
3 comments
Thanks for the detailed response and additional experiments. But the agentic task "Seal-Tools" may be too simple and out-of-date, which makes the result not convincing for me. Could the author providing results on coding task like SWE or knowledge working task like GDPVal under various harness?
We thank the reviewer for the follow-up. To address the concern regarding a harder agentic benchmark, we ran TTRA on a subset of 50 GDPval tasks, using the same models and evaluation harness as in [1]:
| Model | Direct evaluation | After TTRA |
|---|---|---|
| Qwen3-4B-Thinking (base) | 43.0% | 48.4% |
| Agent-R2-4B (fine-tuned variant) | 52.8% | — |
This experiment shows that TTRA is feasible on a more challenging agentic benchmark, such as GDPval. These results are preliminary: TTRA was run for only 50 one-shot steps and 49 TTRL steps, and has not yet been applied to Agent-R2-4B. Since the full experiment cannot be completed within the discussion period, we will include its results in the next version of the paper. Whether or not TTRA narrows the gap between the two models on GDPval, our conclusions on the existing benchmarks remain unaffected, and GDPval provides a valuable extension of our evaluation. We thank the reviewer for this suggestion.
In addition, our paper already includes an agentic task, the conversational setting of MIMIC-CDM (Section 5.1). In this setting, the model is initially given only the patient history and must request diagnostic tests, such as laboratory tests and imaging, one at a time. After each request, it receives the test results from the environment and decides whether to request another test or to give a diagnosis. The task thus involves tool use, multi-turn interaction with an environment, and sequential decisions based on environmental feedback. Under direct evaluation, the ranking of the Qwen3 models on this task does not follow model size, whereas after TTRA it does.
Overall, our evaluation covers a diverse set of benchmarks, including mathematical reasoning, code generation, tool calling, and agentic clinical decision making, among others. Our main conclusions rest on the benchmarks already in the paper, and the GDPval results will extend this coverage to a further agentic setting. We hope the reviewer can take the existing evidence into account.
[1] Pan, Teng, et al. "Agent-R : Scaling Agents for Realistic Productivity Work via Real-Source Data Synthesis." arXiv preprint arXiv:2611.08219 (2026).
We thank the reviewer for the important questions. We address each of the concerns raised on a point-by-point basis below.
W1: Meaning of the score after TTRA TTRA changes the evaluated model, so the score after TTRA does not measure the model's out-of-the-box performance. Instead, it reflects the model's performance after preparation for the task, which [2] refers to as model potential. We use this score as a measure for comparing models, not as an absolute measure of capability, and our conclusions concern such comparisons.
To further support such comparisons, we checked the resulting rankings against pretraining compute, which is fixed before any adaptation and generally predicts performance [2]. For 11 Qwen base models on MATH500, performance does not consistently increase with pretraining compute under direct evaluation, but aligns closely with it after TTRA (Section 3.4).
W2 and Q5: TTRA on agentic tasks We ran new experiments on tool use with Seal-Tools. Under the same TTRA procedure, the average performance gap between three tool-calling models and the models from which they were fine-tuned decreases from 29.0 to 4.0 percentage points (Appendix B.6).
For multi-turn interaction with an environment, we use the conversational setting of MIMIC-CDM, in which the model requests diagnostic tests, receives their results, and then gives a diagnosis. The reward is computed from the final diagnosis of each trajectory, using the majority vote over final diagnoses in the second stage. Under direct evaluation, the ranking of the Qwen3 models does not follow model size, whereas after TTRA it does (Section 5.1).
Q1 We use TTRA as a train-before-test evaluation method. While test-time adaptation typically aims to improve a model’s performance on a given task, we compare models after the same preparation, focusing on their performance gaps and rankings.
Q2: Interpretation of the reduced gap We interpret the reduced gap primarily as elicitation of existing reasoning capabilities, of which format adaptation is one component, rather than as distribution-specific specialization. On the math benchmarks, TTRA uses a single labeled example; all other rewards come from the model's own majority vote. In contrast, the fine-tuned variants we compare against were trained with substantially more external supervision. TTRA therefore relies primarily on the model's own outputs for supervision.
Regarding distribution-specific specialization, our held-out experiment shows that performance is highly consistent between the half of the benchmark used for TTRA and the held-out half (Section 3.3), and the gap between Qwen2.5-Math-1.5B and its Instruct variant remains small even when TTRA is performed on one benchmark and the models are evaluated on another (Appendix A.4).
Regarding language drift, we measured how far each procedure moves the next-token predictions away from those of the base model Qwen2.5-Math-1.5B: on MATH500, 2.1% of token positions have a JS divergence above 0.1 after TTRA, compared with 5.2% for Instruct and 23.5% for DeepSeek-R1-Distill (Appendix B.3).
Q3: TTRA on general-purpose base models We evaluated other model families, such as Llama-3.1-8B (Appendix B.1), and conducted new experiments with the general-purpose base model Qwen2.5-1.5B on MATH500 and HumanEval (Appendix B.4). In both settings, the performance gap between the base models and their fine-tuned variants narrows after TTRA.
Q4: Relation to Yue et al. [1] On reasoning tasks, our conclusions are similar to those of [1], although the two works use different methods. [1] emphasizes large-k answer coverage: whether a correct answer appears among k samples. We measure average accuracy: whether a single response is correct, after all models receive the same preparation. To identify evidence beyond [1], we measured the pass@k of Qwen2.5-1.5B before and after TTRA on the domain-specific benchmark MIMIC-CDM-FI, with random guessing among the four diagnoses as a reference:
Random guessing, which involves no reasoning, achieves 99.0% pass@16, outperforming the LLM on this metric, while its pass@1 is only 25.0%. This illustrates that higher answer coverage does not by itself imply stronger reasoning capability. [1] also discusses this guessing issue for mathematics.
[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.
[2] Zhang, Guanhua, Ricardo Dominguez-Olmedo, and Moritz Hardt. "Train-before-test harmonizes language model rankings." International Conference on Learning Representations. Vol. 2026. 2026.