acceptodds
Under review as a conference paper at ICLR 2027

Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks

10 views

Abstract

Direct evaluation of LLMs on benchmarks can be misleading because different models may have acquired different degrees of familiarity with the task prior to evaluation. The train-before-test approach controls for task familiarity by giving each model task-relevant training before evaluation, originally through supervised fine-tuning(SFT). However, suitable training data is often hard to come by, and evaluation results vary with the data chosen. In this paper, we propose a two-stage test-time reinforcement learning (RL) alignment method for train-before-test. First, RL with a single sample provides a first alignment of the model to the task format, and second, test-time RL with majority-voting reward aligns the model to the benchmark distribution. Our test-time RL alignment method aligns similarly well as the SFT-based train-before-test, but without requiring a task-specific training set. On a domain-specific benchmark without training data, we show that direct evaluation underestimates base models, which perform substantially better once aligned, yielding a comparison of models under the same task preparation. Moreover, for reasoning tasks, base models reach performance comparable to many of their fine-tuned variants after the same label-free test-time alignment, while the fine-tuned variants themselves exhibit only marginal changes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM evaluation, train-before-test, test-time reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7XH2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that direct benchmark scores reflect both model capabilities and familiarity with task formats, output requirements, and data distributions. As a result, base models may be underestimated and gains from fine-tuning may be overstated. To address this confound, the authors propose Test-Time RL Alignment (TTRA). The first stage uses one labeled example for RL-based task-format alignment. The second stage uses benchmark questions and majority-vote-derived rewards for test-time RL adaptation, followed by evaluation. This describes the central pipeline; the code and legal applications use task-specific reward construction. On HumanEval, Qwen2.5-Coder-1.5B Base improves from 3.7% pass@1 under direct evaluation to 70.2% after TTRA, while the Instruct model changes from 70.7% to 73.4%. On MATH500, Qwen2.5-Math-1.5B Base and Instruct achieve 75.50% and 74.54%, respectively, after TTRA, substantially narrowing the gap observed under direct evaluation. The paper also studies clinical and legal tasks using MIMIC-CDM and LEXAM.

Critical strengths

The paper identifies and empirically exposes task-familiarity confounding in benchmark evaluation. The TTRA results show that direct gaps between base and fine-tuned models can partly reflect familiarity with formats and distributions rather than capability alone.

Why this matters for the overall assessment: This problem formulation has direct implications for LLM evaluation practice; even with limited algorithmic novelty, the finding and protocol could justify acceptance.

Critical weaknesses
    • TTRA changes the evaluated model, so the meaning of the resulting score must be defined more carefully. Updating weights on benchmark questions can adapt to templates, distributions, or hidden regularities even without labels; TTRA scores are not equivalent to original zero-shot scores.

    • Why this matters for the overall assessment: If TTRA scores are presented as uncontaminated measures of model capability, the paper has a fundamental interpretational problem. This is the most important conceptual risk to resolve before acceptance.

    • The findings about post-training gains and the applicability of TTRA remain unvalidated in agentic settings. Post-training appears to substantially improve tool use, multi-turn interaction with environments, and long-horizon planning, but the paper does not establish whether the gap between base and post-trained models would still narrow substantially after the same TTRA alignment on these tasks. Success in interactive tasks also depends on sequential decisions and environmental feedback, and it is unclear how effectively the current reward construction and alignment procedure extend to multi-turn trajectories.

    • Why this matters for the overall assessment: This limits the use of the paper’s observations as a general explanation of post-training gains. Without evidence from representative agentic tasks, the results primarily support task-familiarity effects on the evaluated benchmarks; they do not establish that post-training gains in interactive tasks also mainly reflect familiarity alignment. This limitation affects my assessment of the breadth of the conclusions and the generality of the evaluation method.

Questions for authors
  1. Could the authors explicitly define the distinction between TTRA as a train-before-test evaluation protocol and as a test-time adaptation algorithm, and explain which scores should be reported in model comparisons or leaderboards?

  2. Does the reduced Base–Instruct gap reflect format adaptation, elicitation of existing reasoning capabilities, or distribution-specific specialization? Could capability retention, language drift, and transfer to other tasks help distinguish these explanations?

  3. TTRA substantially improves base-model performance on mathematics and code and reduces gap to the fine-tuned variants. Can the findings be reproduced in base models from other model families, particularly general-purpose base models without mathematics- or code-specific specialization?

  4. This paper already cites Yue et al. (2025), Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. Using large-k pass@k, problem coverage, and perplexity analyses, that work finds that, in the settings studied, RLVR gains mainly reflect more efficient sampling of reasoning paths already present in the base model rather than fundamentally new reasoning patterns. The narrowing of the base–fine-tuned performance gap after TTRA is consistent with this observation. What are the central differences and additional conclusions of the present paper relative to that work? In particular, is task-familiarity alignment an empirically distinguishable mechanism from improved sampling efficiency of existing reasoning paths, or an alternative interpretation of the same phenomenon? Could the authors combine pre-/post-TTRA pass@k curves and problem-coverage analyses on matched models and benchmarks to identify evidence beyond the prior work and distinguish changes in task familiarity, sampling efficiency, and reasoning capability boundaries?

  5. Post-training appears to substantially improve base-model capabilities on agentic tasks, such as tool use, multi-turn interaction with environments, and long-horizon planning. Do this paper’s findings about task familiarity and post-training gains, as well as the TTRA method itself, remain applicable and effective in these settings?

3 comments

We thank the reviewer for the important questions. We address each of the concerns raised on a point-by-point basis below.

W1: Meaning of the score after TTRA TTRA changes the evaluated model, so the score after TTRA does not measure the model's out-of-the-box performance. Instead, it reflects the model's performance after preparation for the task, which [2] refers to as model potential. We use this score as a measure for comparing models, not as an absolute measure of capability, and our conclusions concern such comparisons.

To further support such comparisons, we checked the resulting rankings against pretraining compute, which is fixed before any adaptation and generally predicts performance [2]. For 11 Qwen base models on MATH500, performance does not consistently increase with pretraining compute under direct evaluation, but aligns closely with it after TTRA (Section 3.4).

W2 and Q5: TTRA on agentic tasks We ran new experiments on tool use with Seal-Tools. Under the same TTRA procedure, the average performance gap between three tool-calling models and the models from which they were fine-tuned decreases from 29.0 to 4.0 percentage points (Appendix B.6).

For multi-turn interaction with an environment, we use the conversational setting of MIMIC-CDM, in which the model requests diagnostic tests, receives their results, and then gives a diagnosis. The reward is computed from the final diagnosis of each trajectory, using the majority vote over final diagnoses in the second stage. Under direct evaluation, the ranking of the Qwen3 models does not follow model size, whereas after TTRA it does (Section 5.1).

Q1 We use TTRA as a train-before-test evaluation method. While test-time adaptation typically aims to improve a model’s performance on a given task, we compare models after the same preparation, focusing on their performance gaps and rankings.

Q2: Interpretation of the reduced gap We interpret the reduced gap primarily as elicitation of existing reasoning capabilities, of which format adaptation is one component, rather than as distribution-specific specialization. On the math benchmarks, TTRA uses a single labeled example; all other rewards come from the model's own majority vote. In contrast, the fine-tuned variants we compare against were trained with substantially more external supervision. TTRA therefore relies primarily on the model's own outputs for supervision.

Regarding distribution-specific specialization, our held-out experiment shows that performance is highly consistent between the half of the benchmark used for TTRA and the held-out half (Section 3.3), and the gap between Qwen2.5-Math-1.5B and its Instruct variant remains small even when TTRA is performed on one benchmark and the models are evaluated on another (Appendix A.4).

Regarding language drift, we measured how far each procedure moves the next-token predictions away from those of the base model Qwen2.5-Math-1.5B: on MATH500, 2.1% of token positions have a JS divergence above 0.1 after TTRA, compared with 5.2% for Instruct and 23.5% for DeepSeek-R1-Distill (Appendix B.3).

Q3: TTRA on general-purpose base models We evaluated other model families, such as Llama-3.1-8B (Appendix B.1), and conducted new experiments with the general-purpose base model Qwen2.5-1.5B on MATH500 and HumanEval (Appendix B.4). In both settings, the performance gap between the base models and their fine-tuned variants narrows after TTRA.

Q4: Relation to Yue et al. [1] On reasoning tasks, our conclusions are similar to those of [1], although the two works use different methods. [1] emphasizes large-k answer coverage: whether a correct answer appears among k samples. We measure average accuracy: whether a single response is correct, after all models receive the same preparation. To identify evidence beyond [1], we measured the pass@k of Qwen2.5-1.5B before and after TTRA on the domain-specific benchmark MIMIC-CDM-FI, with random guessing among the four diagnoses as a reference:

MIMIC-CDM-FI (%)pass@1pass@4pass@16pass@64
Before TTRA3.512.232.356.5
After TTRA57.461.665.067.2
Random guessing25.068.499.0100.0

Random guessing, which involves no reasoning, achieves 99.0% pass@16, outperforming the LLM on this metric, while its pass@1 is only 25.0%. This illustrates that higher answer coverage does not by itself imply stronger reasoning capability. [1] also discusses this guessing issue for mathematics.

[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.

[2] Zhang, Guanhua, Ricardo Dominguez-Olmedo, and Moritz Hardt. "Train-before-test harmonizes language model rankings." International Conference on Learning Representations. Vol. 2026. 2026.

Thanks for the detailed response and additional experiments. But the agentic task "Seal-Tools" may be too simple and out-of-date, which makes the result not convincing for me. Could the author providing results on coding task like SWE or knowledge working task like GDPVal under various harness?

We thank the reviewer for the follow-up. To address the concern regarding a harder agentic benchmark, we ran TTRA on a subset of 50 GDPval tasks, using the same models and evaluation harness as in [1]:

ModelDirect evaluationAfter TTRA
Qwen3-4B-Thinking (base)43.0%48.4%
Agent-R2-4B (fine-tuned variant)52.8%—

This experiment shows that TTRA is feasible on a more challenging agentic benchmark, such as GDPval. These results are preliminary: TTRA was run for only 50 one-shot steps and 49 TTRL steps, and has not yet been applied to Agent-R2-4B. Since the full experiment cannot be completed within the discussion period, we will include its results in the next version of the paper. Whether or not TTRA narrows the gap between the two models on GDPval, our conclusions on the existing benchmarks remain unaffected, and GDPval provides a valuable extension of our evaluation. We thank the reviewer for this suggestion.

In addition, our paper already includes an agentic task, the conversational setting of MIMIC-CDM (Section 5.1). In this setting, the model is initially given only the patient history and must request diagnostic tests, such as laboratory tests and imaging, one at a time. After each request, it receives the test results from the environment and decides whether to request another test or to give a diagnosis. The task thus involves tool use, multi-turn interaction with an environment, and sequential decisions based on environmental feedback. Under direct evaluation, the ranking of the Qwen3 models on this task does not follow model size, whereas after TTRA it does.

Overall, our evaluation covers a diverse set of benchmarks, including mathematical reasoning, code generation, tool calling, and agentic clinical decision making, among others. Our main conclusions rest on the benchmarks already in the paper, and the GDPval results will extend this coverage to a further agentic setting. We hope the reviewer can take the existing evidence into account.

[1] Pan, Teng, et al. "Agent-R 2^ 2: Scaling Agents for Realistic Productivity Work via Real-Source Data Synthesis." arXiv preprint arXiv:2611.08219 (2026).

Reviewer_wsqiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the impact of task familiarity on LLM benchmark evaluations and raises the Test-Time Reinforcement Learning Alignment (TTRA). TTRA includes two stages: First, a one-shot RL is performed to achieve initial task alignment. Second, pseudo rewards are generated using majority voting to further train the model on test dataset. Experiments covered mathematical reasoning, code generation, clinical diagnosis, and legal reasoning. The results show that TTRA improves consistency in model rankings across relevant benchmarks and significantly narrows the performance gap between base models and fine-tuned models.

Critical strengths
  1. Research question here is very important. The paper provide an interesting perspective and corresponding non-supervised method of solving it. This provides valuable insights to understanding a model’s true capabilities and the benefits of fine-tuning.
  2. The experiment results are impressive. After TTRA, gap between base model and fine-tuned model is hugely decreased. Such as performance of Qwen2.5-Coder-1.5B on HumanEval benchmark is increased from 3.7% to 70.2%, approaching instruction-based fine-tuned models.
  3. The experiments have broad coverage. They involve multiple model sizes and task domains and include ablation studies, held-out evaluations, and cross-benchmark evaluations, which provide substantial support for the key findings.
Critical weaknesses
  1. The core conclusion is lack of sufficient causal evidence. TTRA itself incorporates additional RL training, and the performance improvement may stem from format adaptation, optimization of sampling strategies, or enhanced reasoning capabilities. Current experiments can’t distinguish these factors from task familiarity. Therefore, the conclusion regarding “spurious fine-tuning gains” remains speculative.
  2. The method’s innovation is limited. TTRA primarily combines existing one-shot RL, test-time RL, and train-before-test frameworks. While the application scenario is valuable, it lacks a clear new technical mechanism.
  3. Questions remain regarding the fairness of the evaluation. TTRA updates model parameters using problems from the target benchmark prior to evaluation. Consequently, the resulting scores reflect both model capability and the benefits of additional training. The paper has not sufficiently demonstrated that this evaluation method is more fair. Furthermore, some direct evaluation results are sourced from other papers and may be influenced by differences in prompts and evaluation settings.
  4. Experimental comparisons and generalization validation are not sufficient. There is a lack of comparisons with methods such as few-shot prompting and one-shot supervised fine-tuning (SFT). Stability experiments used different sources of randomness for TTRA and SFT, resulting in comparisons that are not sufficiently rigorous. Some cross-domain experiments also rely on additional annotations or external models, and the generalizability of the conclusions remains to be verified.
Questions for authors
  1. Regarding the core conclusions: How can we prove that the performance gains from TTRA stem primarily from task familiarity, rather than reasoning capabilities derived from additional reinforcement learning or improvements in sampling strategies? Can targeted ablation experiments distinguish between these factors?
  2. Regarding evaluation fairness: Since TTRA updates model parameters using test data, why do the adapted scores more fairly reflect the model’s capabilities? Furthermore, is it possible to revalidate the performance gap between direct evaluation and TTRA under uniform prompt, decoding strategy, and evaluation settings?
  3. Regarding the Adequacy of Experiments: Can baselines such as few-shot prompts and single-sample supervised fine-tuning be added and compared under the same computational budget? Additionally, can stability be reevaluated using a consistent method for controlling randomness, and can the method’s generalization ability across different tasks and supervision conditions be further validated?
4 comments

We thank the reviewer for the important questions. We address each of the concerns raised on a point-by-point basis below.

W1 and Q1: Support for the core conclusion We have removed "spurious" from the titles of Sections 4.1 and 4.2. Since TTRA includes additional RL training, a central question is whether the observed gains involve learning new reasoning abilities. We discuss this question through the following three observations:

First, on the math benchmarks, TTRA uses a single labeled example; all other rewards come from the model's own majority vote. In contrast, the fine-tuned variants we compare against were trained with substantially more external supervision. TTRA therefore relies primarily on the model’s own outputs for supervision.

Second, scores after TTRA follow pretraining compute, which is fixed before any adaptation and generally predicts performance [2, 3]. For 11 Qwen base models on MATH500, performance does not consistently increase with pretraining compute under direct evaluation, but aligns closely with it after TTRA (Section 3.4).

Third, on MATH500, after TTRA the base model performs comparably to some RLVR variants such as Oat-Zero, while DeepSeek-R1-Distill stays ahead of it. This is consistent with [1]: RLVR mainly improves sampling of existing reasoning paths, whereas distillation from a stronger teacher can introduce new reasoning patterns. The same TTRA thus does not bring all models to the same level.

W2: Method novelty We propose TTRA as a train-before-test evaluation method rather than as a new reinforcement learning algorithm. Existing train-before-test evaluation [2, 3] relies on SFT, which requires a task-specific training set that many benchmarks do not have. TTRA requires no such training set, and this is where its novelty lies. Accordingly, we validate it as an evaluation method (Section 3) and then apply it to the benchmarks in Sections 4 and 5.

W3 and Q2: Evaluation fairness TTRA applies the same additional training to every model, and we ask whether the gains reported for fine-tuned models persist after this preparation. Such gains can persist, as discussed in our reply to W1.

The second stage of TTRA uses the test questions but not their labels, in the same way for every model. On a held-out half of the benchmark that was not used for TTRA, the scores are consistent with those on the half that was used (Section 3.3), and the gap between Qwen2.5-Math-1.5B and its Instruct variant remains small when TTRA is performed on one benchmark and the models are evaluated on another (Appendix A.4). This suggests that our results do not come from overfitting to the test questions.

We re-evaluated all 17 models of Section 4.1 on MATH500 with the same prompt, decoding and scoring as in the evaluation after TTRA (Appendix B.2). In this setting, the gap between base and fine-tuned models remains large (Table 6), but narrows substantially after TTRA. We also evaluated these models without the instruction "Let's think step by step and output the final answer within \boxed{}" (Appendix B.2). Removing this instruction alone increases the average gap from +12.0 to +36.2 points for the 1.5B models and from +16.4 to +31.1 points for the 7B models (Table 6). Direct evaluation is therefore not stable either, and its scores do not consistently increase with pretraining compute, as discussed in our reply to W1.

W4 and Q3: Baselines, stability, and generalization As a baseline, we evaluated few-shot prompting on MATH500, giving Qwen2.5-Math-1.5B Base and Instruct the same worked examples from the MATH training split. Instruct led by 17.8–20.2 percentage points across one to four shots, whereas after TTRA the two models scored within one point of each other (Appendix B.5). TTRA also has a lower computational cost than SFT-based train-before-test: on GSM8K, it costs about 6.6 times as much as direct evaluation, compared with about 32 times for SFT (Section 6.2).

Our stability analysis varies the data choice that each method requires in practice: the separate training dataset for SFT-based train-before-test and the single labeled example for TTRA. Although these sources of variation differ, both arise naturally when applying the respective methods. The comparison therefore characterizes how sensitive each method is to its practical data choice (Section 3.2).

In the cross-domain experiments, the comparison is relative within each benchmark. When the setup uses an external model or additional annotations, we hold them fixed across the models being compared.

[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.

[2] Dominguez-Olmedo, Ricardo, Florian Eddie Dorner, and Moritz Hardt. "Training on the test task confounds evaluation and emergence." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Zhang, Guanhua, Ricardo Dominguez-Olmedo, and Moritz Hardt. "Train-before-test harmonizes language model rankings." International Conference on Learning Representations. Vol. 2026. 2026.

Authors' results are convincing, but I still think this could not explain TTRA improves performance through controlling Task Familarity. Since this causal relationship is core and motivation of proposed method, I think it's neccessary to further analyze how has TTRA changed the model's behavior.

We thank the reviewer for the follow-up comments. To address the concern directly, we analyzed how TTRA affects the model’s behavior.

First, we examined format following. Before TTRA, many responses did not follow the task format. On MATH500, TTRA reduces the proportion of responses that end in degenerate repetition from 78.8% to 1.3% for Qwen2.5-1.5B and from 19.6% to 0.4% for Qwen2.5-Math-1.5B. On Seal-Tools, the proportion of Qwen2.5-1.5B-Instruct responses that cannot be parsed as a list of tool calls decreases from 56.9% to 0% after TTRA.

Second, we performed a token-level analysis (Appendix B.3). For each model, we sampled responses to the MATH500 questions and compared its next-token distributions on these responses with those of the base model, Qwen2.5-Math-1.5B. The fraction of token positions with JS divergence above 0.1 is 2.1% after TTRA, compared with 5.2% for Qwen2.5-Math-1.5B-Instruct and 23.5% for DeepSeek-R1-Distill-Qwen-1.5B. After TTRA, the most frequent tokens at these positions are sentence-opening words, such as “To”, “The”, “Let”, and “So”, as well as LaTeX delimiters (word cloud in Figure 14b).

These results indicate that TTRA improves format following while changing the base model’s output distribution only slightly. Following the reviewer’s suggestion, we have also added these analyses to the paper.

Reviewer_mrf5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how task familiarity affects the evaluation of large language models, particularly when comparing base models with their fine-tuned variants. Building on the train-before-test evaluation paradigm, the authors propose Test-Time Reinforcement Learning Alignment (TTRA), a two-stage procedure that combines one-shot reinforcement learning for initial task-format alignment with test-time reinforcement learning using majority-voting-based pseudo-rewards.

The authors evaluate TTRA across mathematical reasoning, code generation, clinical decision-making, and legal reasoning benchmarks. The experiments show that TTRA can substantially improve the performance of base models and reduce the performance gaps between base models and their fine-tuned counterparts. Based on these observations, the paper argues that many apparent fine-tuning gains may largely reflect differences in task familiarity rather than improvements in underlying model capabilities.

Critical strengths
  • Important and interesting research question. The paper investigates a fundamental issue in LLM evaluation: whether performance differences between models reflect underlying capabilities or differences in task familiarity. This is particularly relevant when assessing the effectiveness of post-training methods such as SFT and RLVR.

  • Broad empirical evaluation. The experiments cover multiple model families and task domains, including mathematical reasoning, code generation, clinical decision-making, and legal reasoning. The substantial reduction in performance gaps between base and fine-tuned models is an interesting observation that deserves further investigation.

  • Practical extension of train-before-test evaluation. The proposed approach combines one-shot RL with majority-voting-based test-time RL, reducing the dependence on separately curated task-specific training datasets. The stage-wise ablation and additional held-out and cross-benchmark experiments provide useful evidence regarding the effectiveness of the proposed evaluation procedure.

  • Clear presentation. The paper is generally well organized, with intuitive motivation and extensive experimental visualizations. The main findings are presented clearly, and the implementation details are documented in the appendix.

Critical weaknesses
  • The central conclusion about task familiarity is not sufficiently supported. My primary concern is that the paper interprets the convergence in performance between base and fine-tuned models after TTRA as evidence that the original performance gaps are largely attributable to task familiarity. However, TTRA is itself a reinforcement learning procedure that updates model parameters and optimizes performance on the target task distribution. Consequently, the observed improvements may arise from additional learning, changes in the sampling distribution, or benchmark-specific adaptation, rather than merely correcting for differences in task familiarity. The experiments do not sufficiently disentangle these possibilities. Therefore, the reduced post-TTRA performance gap does not necessarily imply that the models possessed comparable underlying capabilities before adaptation.

  • Applying the same adaptation procedure does not necessarily equalize task familiarity. The paper assumes that applying the same TTRA procedure to different models provides a fair comparison by controlling for task familiarity. However, models may differ substantially in their initial policy distributions, optimization dynamics, learning efficiency, and capacity for further adaptation. In particular, base models may have greater remaining adaptation potential, while fine-tuned models may already be closer to performance saturation. Such differences could naturally lead to a narrowing performance gap even if fine-tuning had genuinely improved the models' capabilities. The paper does not provide an independent definition or measurement of task familiarity to establish that it has actually been equalized across models.

  • Limited methodological novelty. The proposed TTRA framework primarily combines two existing techniques: one-shot reinforcement learning and test-time reinforcement learning with majority-voting-based rewards. Although their combination is reasonable for the train-before-test setting, the paper introduces limited novelty in the underlying reinforcement learning algorithm. The main contribution is therefore better characterized as an evaluation protocol and empirical investigation rather than a new learning methodology. This makes the validity of the empirical interpretation particularly important.

  • Potential confounding from benchmark-specific adaptation. The second stage of TTRA uses questions from the target benchmark to update the model, even though the ground-truth labels are not used. This changes the evaluation setting from measuring the performance of a fixed model to measuring its performance after adaptation to the benchmark distribution. The authors appropriately include held-out and cross-benchmark experiments, which help mitigate concerns about memorization or overfitting to individual test questions. However, these experiments do not fully establish that TTRA isolates task familiarity from additional learning. Instead, they primarily demonstrate that the benefits of adaptation can transfer to related or held-out questions.

  • The reliability of majority-voting-based pseudo-rewards requires further investigation. The second stage relies on the model's own majority-voted answers as pseudo-labels. Such rewards may reinforce systematic errors when incorrect answers dominate the model's output distribution. The paper itself demonstrates a severe failure case in clinical decision-making, where TTRL without initial alignment collapses toward predicting a single pathology. Although the one-shot stage alleviates this issue, the paper does not sufficiently characterize the conditions under which the pseudo-rewards remain reliable or how their quality affects the resulting model comparisons.

  • Some claims are stronger than the experimental evidence warrants. The results convincingly demonstrate that base models can achieve substantial improvements through relatively lightweight test-time adaptation. However, this does not establish that the original gains from SFT or RLVR were spurious or that fine-tuning failed to improve reasoning capabilities. A more cautious interpretation would be that the measured advantages of fine-tuned models depend substantially on the evaluation and adaptation protocol. This distinction is important because the paper's broader conclusions about post-training effectiveness rely on interpreting post-adaptation performance as evidence of pre-adaptation capability.

Questions for authors

Please refer to the Weakness.

2 comments

We thank the reviewer for the important questions. We address each of the concerns raised on a point-by-point basis below.

W1: Support for the central conclusion TTRA is a train-before-test evaluation method [2, 3] that applies an RL-based preparation procedure to a model before evaluation to measure its model potential [3]. We agree that this preparation updates model parameters. The reviewer’s main concern is that this preparation teaches it new ability, and our claim is that it mainly elicits the ability the base model already has. Two observations support the latter:

The main observation is that TTRA is given very little to learn from, yet the gains are large and transfer:

  • Our held-out experiment from Section 3.3 shows that performance is highly consistent between the half of the benchmark used for TTRA and the held-out half (Section 3.3), and the base–Instruct gap remains small even when TTRA is performed on a different benchmark (Appendix A.4; see our reply to W4). These results suggest that the reduced gap reflects transferable task preparation rather than adaptation specific to the evaluated benchmark.
  • Moreover, on the math benchmarks, TTRA uses a single labeled example; all other rewards come from the model's own majority vote. In contrast, the fine-tuned variants we compare against were trained with substantially more external supervision. TTRA therefore relies primarily on the model’s own outputs rather than external supervision.

Second, scores after TTRA follow pretraining compute. We have added a new experiment to Section 3.4. In general, we expect a model with more pretraining compute to perform better, as discussed in [2, 3]. We evaluated some models on MATH500 and examined how their performance relates to pretraining compute. Under direct evaluation, performance does not consistently increase with pretraining compute. Interestingly, after TTRA, performance aligns closely with pretraining compute, which is fixed before adaptation.

W2: Differences between models under the same TTRA For our reasoning experiments, several RLVR variants end at a level similar to that of the base model, but after the same TTRA, DeepSeek-R1-Distill-Qwen-1.5B still performs better than its base model (Section 4.1). This is consistent with [1], which reports that distillation from a much stronger teacher can introduce new reasoning patterns. TTRA gives every model the same preparation but does not necessarily equalize the models in every respect; our conclusions concern the comparison after this preparation.

W3: Method novelty We propose TTRA as a train-before-test evaluation method rather than as a new reinforcement learning algorithm. Existing train-before-test evaluation [2, 3] relies on SFT, which requires a task-specific training set that many benchmarks do not have. TTRA requires no such training set, and this is where its novelty lies. Accordingly, we validate it as an evaluation method (Section 3) and then apply it to the benchmarks in Sections 4 and 5.

W4: Benchmark-specific adaptation TTRA measures the performance of a model after the same preparation as all other models, rather than its out-of-the-box performance. Even when TTRA is performed on one benchmark and the models are evaluated on another, the gap between Qwen2.5-Math-1.5B and its Instruct variant remains small (Appendix A.4). The reduced gap therefore does not appear to depend strongly on adaptation to the evaluated benchmark. Moreover, as discussed in our reply to W1, the scores after TTRA follow pretraining compute, which is fixed before adaptation.

W5: Reliability of majority-vote rewards Regarding the conditions under which the pseudo-rewards remain reliable, majority-vote rewards depend on the accuracy of the model's own answers, which is what motivates the one-shot stage of TTRA. The clinical failure case was obtained without this stage; with both stages, we observed no such collapse in the settings reported in the paper.

Regarding how the quality of the pseudo-labels affects the resulting model comparisons, this quality differs between models, because each model's labels come from its own majority vote. Variants such as Oat-Zero start from higher accuracy and can therefore be expected to learn from more reliable labels than the base model, yet they gain little from TTRA, while the base model gains substantially and reaches their level (Section 4.1). This suggests that the narrowed gap is unlikely to be explained by better pseudo-labels for the base model.

W6: Strength of the claims We agree that the measured advantages of fine-tuned models depend substantially on the evaluation and adaptation method. TTRA compares the performance of models after the same preparation, not their out-of-the-box performance. We have removed "spurious" from the titles of Sections 4.1 and 4.2. Our replies to W1 and W2 discuss the interpretation of the scores after TTRA.

[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.

[2] Dominguez-Olmedo, Ricardo, Florian Eddie Dorner, and Moritz Hardt. "Training on the test task confounds evaluation and emergence." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Zhang, Guanhua, Ricardo Dominguez-Olmedo, and Moritz Hardt. "Train-before-test harmonizes language model rankings." International Conference on Learning Representations. Vol. 2026. 2026.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period is coming to an end, I kindly encourage you to review and respond to the authors' rebuttal as soon as possible.

Thank you very much for your time and valuable contributions to the review process.

Best regards,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.